2024-09-21T01:09:48.562 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T01:09:48.579 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T01:09:48.652 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367 branch: wip-pdonnell-testing-20240920.212106-debug description: fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench conf/{client mds mgr mon osd} distro/single-container-host overrides/{ignorelist_health pg_health}} email: pdonnell@redhat.com first_in_suite: false job_id: '7913367' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240920.212106-debug ceph: cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 2 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: client mount timeout: 30 debug client: 20 debug mgr: 20 debug ms: 1 mon warn on pool no app: false mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - fs.*is degraded - filesystem is degraded - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - filesystem is offline - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - insufficient standby MDS daemons available - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 1843221895adccfb3b692060251dd6a0183bc410 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 1843221895adccfb3b692060251dd6a0183bc410 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-pdonnell-testing-20240920.212106-debug sha1: 1843221895adccfb3b692060251dd6a0183bc410 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 7864 sha1: 1843221895adccfb3b692060251dd6a0183bc410 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240920.212106-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1843221895adccfb3b692060251dd6a0183bc410 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOrbSXLaRvcdfXXRkrhL4b+mDPY1eZwFiGJaUE3JFTZKL8p/ku9/Lq16lmcXvxZ/i13JptoqFgMwIUFqIwBLSc= smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDLfm3N02bwhpw/wtW0Gr4ZEiXsHb8vO5RP2z4LLAgt+I+dgdwsnOpHKVvaFHMakD3YCOqpFyVM1ngnVwqwQmA= tasks: - install: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create foo - ceph fs volume create bar - fs.ready: timeout: 300 - ceph-fuse: client.0: cephfs_name: foo - ceph-fuse: client.1: cephfs_name: bar - workunit: clients: all: - suites/dbench.sh - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_23:48:22 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T01:09:48.653 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa; will attempt to use it 2024-09-21T01:09:48.653 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1843221895adccfb3b692060251dd6a0183bc410/qa/tasks 2024-09-21T01:09:48.653 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T01:09:48.655 INFO:teuthology.task.internal:Checking packages... 2024-09-21T01:09:48.674 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1843221895adccfb3b692060251dd6a0183bc410' 2024-09-21T01:09:48.674 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T01:09:48.674 INFO:teuthology.packaging:ref: None 2024-09-21T01:09:48.674 INFO:teuthology.packaging:tag: None 2024-09-21T01:09:48.674 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:09:48.674 INFO:teuthology.packaging:sha1: 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:09:48.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:09:48.822 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5120.g18432218 2024-09-21T01:09:48.823 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T01:09:48.832 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T01:09:48.832 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T01:09:48.867 INFO:teuthology.task.internal:Saving configuration 2024-09-21T01:09:48.880 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T01:09:48.889 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T01:09:48.909 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 01:04:02.921942', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEOrbSXLaRvcdfXXRkrhL4b+mDPY1eZwFiGJaUE3JFTZKL8p/ku9/Lq16lmcXvxZ/i13JptoqFgMwIUFqIwBLSc='} 2024-09-21T01:09:48.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 01:04:02.923509', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDLfm3N02bwhpw/wtW0Gr4ZEiXsHb8vO5RP2z4LLAgt+I+dgdwsnOpHKVvaFHMakD3YCOqpFyVM1ngnVwqwQmA='} 2024-09-21T01:09:48.927 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T01:09:48.935 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-21T01:09:48.935 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-21T01:09:48.935 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T01:09:48.982 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd8ecc60160>, signals=[15]) 2024-09-21T01:09:48.982 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T01:09:48.990 INFO:teuthology.task.internal:Opening connections... 2024-09-21T01:09:48.990 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-09-21T01:09:48.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:09:49.066 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-09-21T01:09:49.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:09:49.139 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T01:09:49.152 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-09-21T01:09:49.171 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-09-21T01:09:49.171 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-09-21T01:09:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-09-21T01:09:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="9" 2024-09-21T01:09:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-09-21T01:09:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-09-21T01:09:49.229 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="9" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el9" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:LOGO="fedora-logo-icon" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T01:09:49.230 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T01:09:49.231 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-09-21T01:09:49.254 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-09-21T01:09:49.271 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-09-21T01:09:49.271 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-09-21T01:09:49.327 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-09-21T01:09:49.327 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T01:09:49.328 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T01:09:49.329 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-09-21T01:09:49.354 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T01:09:49.362 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T01:09:49.368 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T01:09:49.368 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-09-21T01:09:49.371 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-09-21T01:09:49.386 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T01:09:49.392 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T01:09:49.392 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-09-21T01:09:49.429 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-09-21T01:09:49.455 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T01:09:49.502 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T01:09:49.517 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T01:09:49.518 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T01:09:49.518 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T01:09:49.518 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T01:09:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:09:49.534 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-09-21T01:09:49.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:09:49.535 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-09-21T01:09:49.590 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-503.el9.x86_64 2024-09-21T01:09:49.590 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-503.el9.x86_64 2024-09-21T01:09:49.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-09-21T01:09:49.593 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-503.el9.x86_64 2024-09-21T01:09:49.593 INFO:teuthology.task.kernel:Running kernel on smithi045: 5.14.0-503.el9.x86_64 2024-09-21T01:09:49.593 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-09-21T01:09:52.034 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-21T01:09:53.066 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.3 MB 00:01 2024-09-21T01:09:55.109 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-21T01:10:00.601 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-21T01:10:00.653 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-21T01:10:02.228 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 19 kB 00:00 2024-09-21T01:10:03.658 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-21T01:10:05.609 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 8.0 MB/s | 6.5 MB 00:00 2024-09-21T01:10:07.085 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 19 kB 00:00 2024-09-21T01:10:08.667 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-21T01:10:09.165 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T01:10:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T01:10:11.031 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T01:10:11.138 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T01:10:11.152 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 79 M 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 126 M 2024-09-21T01:10:11.153 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:10:11.368 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 409 kB/s | 36 kB 00:00 2024-09-21T01:10:11.785 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-21T01:10:12.302 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 34 MB/s | 35 MB 00:01 2024-09-21T01:10:12.619 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 23 MB/s | 29 MB 00:01 2024-09-21T01:10:12.619 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:10:12.619 INFO:teuthology.orchestra.run.smithi090.stdout:Total 54 MB/s | 79 MB 00:01 2024-09-21T01:10:13.302 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:10:13.669 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:10:13.669 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:10:14.295 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T01:10:14.574 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:10:14.574 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:10:16.268 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T01:10:16.268 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T01:10:16.367 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:10:16.371 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-09-21T01:10:16.372 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:10:16.373 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 79 M 2024-09-21T01:10:16.373 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 126 M 2024-09-21T01:10:16.373 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:10:16.913 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 87 kB/s | 36 kB 00:00 2024-09-21T01:10:17.037 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:10:17.741 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:10:17.867 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:10:20.263 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 8.5 MB/s | 29 MB 00:03 2024-09-21T01:10:20.637 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:10:20.795 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:10:21.138 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.4 MB/s | 16 MB 00:04 2024-09-21T01:10:25.238 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 4.0 MB/s | 35 MB 00:08 2024-09-21T01:10:25.238 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:10:25.238 INFO:teuthology.orchestra.run.smithi045.stdout:Total 8.9 MB/s | 79 MB 00:08 2024-09-21T01:10:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:10:25.688 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:10:25.953 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:10:26.329 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:10:26.329 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:10:27.313 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:10:27.314 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:10:29.874 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:10:30.504 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:10:30.576 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:10:30.657 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:10:33.406 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:10:33.534 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:10:38.691 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:10:38.734 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:10:43.871 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:10.160 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:10.199 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:10.587 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:10.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:11:10.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:11:10.587 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:11:11.483 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:11:11.484 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:11:11.877 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:11:11.985 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:11:11.985 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:11:11.985 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T01:11:11.986 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T01:11:11.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T01:11:11.986 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-09-21T01:11:11.986 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T01:11:11.986 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:11:12.044 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:11:12.045 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:11:12.045 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 82.el9 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-09-21T01:11:12.076 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8274670 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T01:11:12.077 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T01:11:12.078 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T01:11:12.078 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-09-21T01:11:12.078 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:11:12.078 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T01:11:12.078 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T01:11:12.079 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-508.el9.x86_64 2024-09-21T01:11:12.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T01:11:12.750 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-09-21T01:11:14.674 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T01:11:14.701 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-09-21T01:11:14.703 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-09-21T01:11:14.726 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.l5o3oJQ1Jo 2024-09-21T01:11:14.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.l5o3oJQ1Jo 2024-09-21T01:11:14.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.l5o3oJQ1Jo 2024-09-21T01:11:14.938 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.l5o3oJQ1Jo is 6KB 2024-09-21T01:11:14.986 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.l5o3oJQ1Jo 2024-09-21T01:11:15.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T01:11:15.072 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T01:11:15.072 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T01:11:15.072 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T01:11:15.072 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T01:11:15.074 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T01:11:15.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-09-21T01:11:22.987 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:23.028 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:23.518 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:23.518 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T01:11:23.518 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T01:11:23.518 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:11:24.365 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:11:24.791 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:11:24.880 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:11:24.880 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:11:24.880 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T01:11:24.881 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T01:11:24.881 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T01:11:24.881 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-09-21T01:11:24.881 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T01:11:24.881 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T01:11:24.939 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T01:11:24.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T01:11:24.940 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.06 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 82.el9 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Group : Unspecified 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 8274670 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T01:11:24.974 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : builder@centos.org 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-09-21T01:11:24.975 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:11:24.976 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T01:11:24.976 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T01:11:24.976 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 5.14.0-508.el9.x86_64 2024-09-21T01:11:24.976 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T01:11:25.754 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-09-21T01:11:27.157 INFO:teuthology.orchestra.run.smithi045.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T01:11:27.179 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-09-21T01:11:27.181 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-09-21T01:11:27.196 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.jYCMtxdm75 2024-09-21T01:11:27.196 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jYCMtxdm75 2024-09-21T01:11:27.262 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.jYCMtxdm75 2024-09-21T01:11:27.402 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.jYCMtxdm75 is 6KB 2024-09-21T01:11:27.450 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.jYCMtxdm75 2024-09-21T01:11:27.465 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T01:11:27.527 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T01:11:27.527 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T01:11:27.527 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T01:11:27.527 INFO:teuthology.orchestra.run.smithi045.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T01:11:27.529 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T01:11:27.801 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-09-21T01:11:45.249 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-09-21T01:11:45.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:11:45.250 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:11:57.817 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-09-21T01:11:57.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-21T01:11:57.818 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:12:03.567 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-09-21T01:12:12.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:12:12.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:12:15.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-09-21T01:12:16.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-09-21T01:12:25.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-21T01:12:25.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:12:27.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:12:27.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:12:30.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-09-21T01:12:45.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:12:45.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:13:04.239 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-09-21T01:13:22.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:13:22.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:13:22.728 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-09-21T01:13:23.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:13:23.226 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T01:13:23.226 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-09-21T01:13:23.240 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-508.el9.x86_64 2024-09-21T01:13:23.241 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T01:13:23.241 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T01:13:23.241 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T01:13:24.242 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-09-21T01:13:24.242 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T01:13:24.822 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-09-21T01:13:24.841 DEBUG:teuthology.parallel:result is None 2024-09-21T01:13:25.377 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T01:13:37.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-21T01:13:37.379 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:13:37.716 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-09-21T01:13:38.078 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-21T01:13:38.079 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T01:13:38.079 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-09-21T01:13:38.096 INFO:teuthology.orchestra.run.smithi045.stdout:5.14.0-508.el9.x86_64 2024-09-21T01:13:38.096 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T01:13:38.096 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T01:13:38.096 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T01:13:39.096 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-09-21T01:13:39.097 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T01:13:39.179 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-09-21T01:13:39.200 DEBUG:teuthology.parallel:result is None 2024-09-21T01:13:39.200 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T01:13:39.207 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T01:13:39.207 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T01:13:39.209 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T01:13:39.228 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T01:13:39.234 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T01:13:39.244 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T01:13:39.244 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T01:13:39.268 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T01:13:39.303 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T01:13:39.310 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T01:13:39.311 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T01:13:39.330 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T01:13:39.366 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:13:39.368 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:13:39.378 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:13:39.380 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T01:13:39.381 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T01:13:39.388 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T01:13:39.388 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T01:13:39.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T01:13:39.449 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T01:13:39.457 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T01:13:39.457 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T01:13:39.491 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T01:13:39.508 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:13:39.591 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:13:39.645 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:13:39.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:13:39.774 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:13:39.774 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T01:13:39.838 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:13:39.879 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T01:13:39.940 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:13:40.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T01:13:40.067 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:13:40.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T01:13:40.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-09-21T01:13:40.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-09-21T01:13:40.185 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:13:40.227 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T01:13:40.534 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T01:13:40.541 INFO:teuthology.task.internal:Starting timer... 2024-09-21T01:13:40.542 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T01:13:40.558 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T01:13:40.568 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-21T01:13:40.568 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-09-21T01:13:40.627 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-09-21T01:13:40.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-09-21T01:13:40.696 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-09-21T01:13:40.697 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T01:13:40.698 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-09-21T01:13:40.720 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-09-21T01:13:40.721 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-09-21T01:13:40.747 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-09-21T01:13:40.747 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi090.front.sepia.ceph.com': 'permissive'} 2024-09-21T01:13:40.748 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T01:13:40.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:13:40.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T01:13:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:13:40.815 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T01:13:40.815 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-09-21T01:13:40.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-09-21T01:13:40.894 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T01:13:40.903 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T01:13:40.914 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T01:13:40.915 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi090.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T01:17:26.275 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi090.front.sepia.ceph.com')] 2024-09-21T01:17:26.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-21T01:17:26.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:17:26.361 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-09-21T01:17:26.447 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-09-21T01:17:26.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:17:26.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T01:17:26.549 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-09-21T01:17:26.630 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-21T01:17:26.630 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T01:17:26.639 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T01:17:26.640 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T01:17:26.640 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:17:26.643 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T01:17:26.643 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T01:17:26.673 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T01:17:26.688 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T01:17:26.699 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T01:17:26.714 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T01:17:26.715 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-09-21T01:17:26.726 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-09-21T01:17:26.737 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-09-21T01:17:26.741 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T01:17:26.748 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-09-21T01:17:26.756 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T01:17:26.762 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T01:17:26.776 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T01:17:26.822 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:17:26.825 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:17:26.826 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-09-21T01:17:26.826 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.826 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.826 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.826 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.836 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.839 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T01:17:26.840 INFO:teuthology.run_tasks:Running task install... 2024-09-21T01:17:26.851 DEBUG:teuthology.task.install:project ceph 2024-09-21T01:17:26.852 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'}} 2024-09-21T01:17:26.852 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T01:17:26.852 INFO:teuthology.task.install:Using flavor: default 2024-09-21T01:17:26.857 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T01:17:26.858 INFO:teuthology.task.install:extra packages: [] 2024-09-21T01:17:26.858 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T01:17:26.859 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:17:26.860 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'tag': None, 'wait_for_package': False} 2024-09-21T01:17:26.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:17:27.009 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T01:17:27.009 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T01:17:27.013 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/ 2024-09-21T01:17:27.013 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5120.g18432218 2024-09-21T01:17:27.117 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T01:17:27.117 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:17:27.117 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T01:17:27.145 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T01:17:27.146 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T01:17:27.151 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T01:17:27.152 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:17:27.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T01:17:27.178 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T01:17:27.179 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1843221895adccfb3b692060251dd6a0183bc410/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T01:17:27.212 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T01:17:27.244 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T01:17:27.288 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T01:17:27.319 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T01:17:27.354 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-09-21T01:17:27.356 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-09-21T01:17:27.383 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-09-21T01:17:27.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-09-21T01:17:27.606 INFO:teuthology.orchestra.run.smithi045.stdout:45 files removed 2024-09-21T01:17:27.614 INFO:teuthology.orchestra.run.smithi090.stdout:45 files removed 2024-09-21T01:17:27.633 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-09-21T01:17:27.641 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-09-21T01:17:28.310 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 219 kB/s | 84 kB 00:00 2024-09-21T01:17:28.313 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 220 kB/s | 84 kB 00:00 2024-09-21T01:17:28.603 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 97 kB/s | 24 kB 00:00 2024-09-21T01:17:28.614 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 95 kB/s | 24 kB 00:00 2024-09-21T01:17:28.871 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-21T01:17:28.898 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-21T01:17:29.465 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-21T01:17:29.479 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T01:17:31.517 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-21T01:17:36.567 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - CRB 6.0 MB/s | 6.5 MB 00:01 2024-09-21T01:17:38.026 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 9 - Extras packages 101 kB/s | 19 kB 00:00 2024-09-21T01:17:41.090 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 1.9 MB/s | 20 MB 00:10 2024-09-21T01:17:43.319 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 23 MB 00:05 2024-09-21T01:17:47.550 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-21T01:17:48.846 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-21T01:17:49.252 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 19 kB 00:00 2024-09-21T01:17:50.465 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-21T01:17:51.201 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:17:51.220 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T01:17:51.221 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T01:17:51.222 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:17:51.223 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:17:51.224 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:17:51.224 INFO:teuthology.orchestra.run.smithi045.stdout:Install 31 Packages 2024-09-21T01:17:51.224 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-09-21T01:17:51.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:17:51.224 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 59 M 2024-09-21T01:17:51.224 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:17:51.989 INFO:teuthology.orchestra.run.smithi045.stdout:(1/33): ceph-base-19.3.0-5120.g18432218.el9.x86 8.1 MB/s | 5.5 MB 00:00 2024-09-21T01:17:52.031 INFO:teuthology.orchestra.run.smithi045.stdout:(2/33): ceph-selinux-19.3.0-5120.g18432218.el9. 606 kB/s | 25 kB 00:00 2024-09-21T01:17:52.183 INFO:teuthology.orchestra.run.smithi045.stdout:(3/33): ceph-radosgw-19.3.0-5120.g18432218.el9. 12 MB/s | 10 MB 00:00 2024-09-21T01:17:52.190 INFO:teuthology.orchestra.run.smithi045.stdout:(4/33): libcephfs2-19.3.0-5120.g18432218.el9.x8 4.7 MB/s | 766 kB 00:00 2024-09-21T01:17:52.223 INFO:teuthology.orchestra.run.smithi045.stdout:(5/33): libradosstriper1-19.3.0-5120.g18432218. 6.6 MB/s | 507 kB 00:00 2024-09-21T01:17:52.274 INFO:teuthology.orchestra.run.smithi045.stdout:(6/33): python3-ceph-argparse-19.3.0-5120.g1843 902 kB/s | 45 kB 00:00 2024-09-21T01:17:52.341 INFO:teuthology.orchestra.run.smithi045.stdout:(7/33): python3-ceph-common-19.3.0-5120.g184322 2.2 MB/s | 148 kB 00:00 2024-09-21T01:17:52.575 INFO:teuthology.orchestra.run.smithi045.stdout:(8/33): ceph-common-19.3.0-5120.g18432218.el9.x 16 MB/s | 20 MB 00:01 2024-09-21T01:17:52.600 INFO:teuthology.orchestra.run.smithi045.stdout:(9/33): python3-cephfs-19.3.0-5120.g18432218.el 624 kB/s | 161 kB 00:00 2024-09-21T01:17:52.625 INFO:teuthology.orchestra.run.smithi045.stdout:(10/33): python3-rados-19.3.0-5120.g18432218.el 6.3 MB/s | 321 kB 00:00 2024-09-21T01:17:52.650 INFO:teuthology.orchestra.run.smithi045.stdout:(11/33): python3-rbd-19.3.0-5120.g18432218.el9. 5.9 MB/s | 307 kB 00:00 2024-09-21T01:17:52.676 INFO:teuthology.orchestra.run.smithi045.stdout:(12/33): python3-rgw-19.3.0-5120.g18432218.el9. 1.9 MB/s | 99 kB 00:00 2024-09-21T01:17:52.751 INFO:teuthology.orchestra.run.smithi045.stdout:(13/33): librgw2-19.3.0-5120.g18432218.el9.x86_ 9.4 MB/s | 5.3 MB 00:00 2024-09-21T01:17:52.793 INFO:teuthology.orchestra.run.smithi045.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 752 kB/s | 107 kB 00:00 2024-09-21T01:17:52.843 INFO:teuthology.orchestra.run.smithi045.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-09-21T01:17:52.868 INFO:teuthology.orchestra.run.smithi045.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 885 kB/s | 171 kB 00:00 2024-09-21T01:17:52.894 INFO:teuthology.orchestra.run.smithi045.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 160 kB 00:00 2024-09-21T01:17:52.944 INFO:teuthology.orchestra.run.smithi045.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-21T01:17:52.970 INFO:teuthology.orchestra.run.smithi045.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-09-21T01:17:53.003 INFO:teuthology.orchestra.run.smithi045.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-21T01:17:53.028 INFO:teuthology.orchestra.run.smithi045.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-21T01:17:53.054 INFO:teuthology.orchestra.run.smithi045.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 944 kB/s | 47 kB 00:00 2024-09-21T01:17:53.079 INFO:teuthology.orchestra.run.smithi045.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-21T01:17:53.138 INFO:teuthology.orchestra.run.smithi045.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-21T01:17:53.163 INFO:teuthology.orchestra.run.smithi045.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 385 kB/s | 26 kB 00:00 2024-09-21T01:17:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 826 kB/s | 49 kB 00:00 2024-09-21T01:17:53.222 INFO:teuthology.orchestra.run.smithi045.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-21T01:17:53.256 INFO:teuthology.orchestra.run.smithi045.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-21T01:17:53.306 INFO:teuthology.orchestra.run.smithi045.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-21T01:17:53.365 INFO:teuthology.orchestra.run.smithi045.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T01:17:53.457 INFO:teuthology.orchestra.run.smithi045.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T01:17:53.549 INFO:teuthology.orchestra.run.smithi045.stdout:(32/33): librados2-19.3.0-5120.g18432218.el9.x8 14 MB/s | 3.4 MB 00:00 2024-09-21T01:17:53.641 INFO:teuthology.orchestra.run.smithi045.stdout:(33/33): librbd1-19.3.0-5120.g18432218.el9.x86_ 11 MB/s | 3.2 MB 00:00 2024-09-21T01:17:53.646 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:17:53.646 INFO:teuthology.orchestra.run.smithi045.stdout:Total 24 MB/s | 59 MB 00:02 2024-09-21T01:17:53.761 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:17:53.802 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:17:53.802 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:17:54.500 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:17:54.500 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:17:55.914 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T01:17:56.040 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:17:56.125 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T01:17:56.154 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T01:17:56.179 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T01:17:56.210 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T01:17:56.231 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T01:17:56.421 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T01:17:56.475 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T01:17:56.884 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T01:17:56.957 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 7/35 2024-09-21T01:17:57.051 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 7/35 2024-09-21T01:17:57.144 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 8/35 2024-09-21T01:17:57.175 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/35 2024-09-21T01:17:57.238 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/35 2024-09-21T01:17:57.269 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T01:17:57.311 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el9. 11/35 2024-09-21T01:17:57.403 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 12/35 2024-09-21T01:17:57.419 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 13/35 2024-09-21T01:17:57.467 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 13/35 2024-09-21T01:17:57.510 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T01:17:57.621 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T01:17:57.677 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T01:17:57.721 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T01:17:57.752 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T01:17:57.780 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T01:17:57.814 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T01:17:57.854 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T01:17:58.212 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T01:17:58.249 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:17:58.260 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 23/35 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 10 M 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-09-21T01:17:58.266 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.4 M 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.2 M 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.5 M 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 20 M 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:19.3.0-5120.g18432218.el9 ceph 25 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 766 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T01:17:58.267 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:19.3.0-5120.g18432218.el9 ceph 507 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.3 M 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:19.3.0-5120.g18432218.el9 ceph 45 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:19.3.0-5120.g18432218.el9 ceph 161 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:19.3.0-5120.g18432218.el9 ceph 321 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 307 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:19.3.0-5120.g18432218.el9 ceph 99 k 2024-09-21T01:17:58.268 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout:Installing weak dependencies: 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout:Install 31 Packages 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-09-21T01:17:58.269 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:17:58.270 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 59 M 2024-09-21T01:17:58.270 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:17:58.356 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 23/35 2024-09-21T01:17:58.382 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 24/35 2024-09-21T01:17:58.427 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T01:17:58.954 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T01:17:58.980 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 27/35 2024-09-21T01:17:59.052 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 27/35 2024-09-21T01:17:59.129 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 28/35 2024-09-21T01:17:59.166 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9.x8 29/35 2024-09-21T01:17:59.233 INFO:teuthology.orchestra.run.smithi090.stdout:(1/33): ceph-base-19.3.0-5120.g18432218.el9.x86 9.2 MB/s | 5.5 MB 00:00 2024-09-21T01:17:59.300 INFO:teuthology.orchestra.run.smithi090.stdout:(2/33): ceph-selinux-19.3.0-5120.g18432218.el9. 378 kB/s | 25 kB 00:00 2024-09-21T01:17:59.434 INFO:teuthology.orchestra.run.smithi090.stdout:(3/33): ceph-radosgw-19.3.0-5120.g18432218.el9. 13 MB/s | 10 MB 00:00 2024-09-21T01:17:59.467 INFO:teuthology.orchestra.run.smithi090.stdout:(4/33): libcephfs2-19.3.0-5120.g18432218.el9.x8 4.5 MB/s | 766 kB 00:00 2024-09-21T01:17:59.501 INFO:teuthology.orchestra.run.smithi090.stdout:(5/33): libradosstriper1-19.3.0-5120.g18432218. 7.4 MB/s | 507 kB 00:00 2024-09-21T01:17:59.568 INFO:teuthology.orchestra.run.smithi090.stdout:(6/33): python3-ceph-argparse-19.3.0-5120.g1843 676 kB/s | 45 kB 00:00 2024-09-21T01:17:59.661 INFO:teuthology.orchestra.run.smithi090.stdout:(7/33): python3-ceph-common-19.3.0-5120.g184322 1.6 MB/s | 148 kB 00:00 2024-09-21T01:17:59.887 INFO:teuthology.orchestra.run.smithi090.stdout:(8/33): ceph-common-19.3.0-5120.g18432218.el9.x 16 MB/s | 20 MB 00:01 2024-09-21T01:17:59.912 INFO:teuthology.orchestra.run.smithi090.stdout:(9/33): python3-cephfs-19.3.0-5120.g18432218.el 645 kB/s | 161 kB 00:00 2024-09-21T01:17:59.946 INFO:teuthology.orchestra.run.smithi090.stdout:(10/33): python3-rados-19.3.0-5120.g18432218.el 5.4 MB/s | 321 kB 00:00 2024-09-21T01:18:00.038 INFO:teuthology.orchestra.run.smithi090.stdout:(11/33): librgw2-19.3.0-5120.g18432218.el9.x86_ 9.3 MB/s | 5.3 MB 00:00 2024-09-21T01:18:00.063 INFO:teuthology.orchestra.run.smithi090.stdout:(12/33): python3-rgw-19.3.0-5120.g18432218.el9. 849 kB/s | 99 kB 00:00 2024-09-21T01:18:00.088 INFO:teuthology.orchestra.run.smithi090.stdout:(13/33): python3-rbd-19.3.0-5120.g18432218.el9. 1.7 MB/s | 307 kB 00:00 2024-09-21T01:18:00.488 INFO:teuthology.orchestra.run.smithi090.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 237 kB/s | 107 kB 00:00 2024-09-21T01:18:00.597 INFO:teuthology.orchestra.run.smithi090.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-21T01:18:00.623 INFO:teuthology.orchestra.run.smithi090.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 305 kB/s | 171 kB 00:00 2024-09-21T01:18:00.648 INFO:teuthology.orchestra.run.smithi090.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 286 kB/s | 160 kB 00:00 2024-09-21T01:18:00.765 INFO:teuthology.orchestra.run.smithi090.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-09-21T01:18:00.824 INFO:teuthology.orchestra.run.smithi090.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-21T01:18:00.941 INFO:teuthology.orchestra.run.smithi090.stdout:(20/33): qatlib-service-24.02.0-1.el9.x86_64.rp 307 kB/s | 36 kB 00:00 2024-09-21T01:18:00.974 INFO:teuthology.orchestra.run.smithi090.stdout:(21/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.7 MB/s | 662 kB 00:00 2024-09-21T01:18:01.024 INFO:teuthology.orchestra.run.smithi090.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.0 MB/s | 308 kB 00:00 2024-09-21T01:18:01.050 INFO:teuthology.orchestra.run.smithi090.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 775 kB/s | 221 kB 00:00 2024-09-21T01:18:01.075 INFO:teuthology.orchestra.run.smithi090.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 353 kB/s | 47 kB 00:00 2024-09-21T01:18:01.125 INFO:teuthology.orchestra.run.smithi090.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-09-21T01:18:01.151 INFO:teuthology.orchestra.run.smithi090.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.8 MB/s | 67 kB 00:00 2024-09-21T01:18:01.176 INFO:teuthology.orchestra.run.smithi090.stdout:(27/33): liboath-2.6.7-2.el9.x86_64.rpm 520 kB/s | 49 kB 00:00 2024-09-21T01:18:01.218 INFO:teuthology.orchestra.run.smithi090.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-21T01:18:01.277 INFO:teuthology.orchestra.run.smithi090.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.5 MB/s | 839 kB 00:00 2024-09-21T01:18:01.362 INFO:teuthology.orchestra.run.smithi090.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-21T01:18:01.412 INFO:teuthology.orchestra.run.smithi090.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-09-21T01:18:01.541 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 30/35 2024-09-21T01:18:01.562 INFO:teuthology.orchestra.run.smithi090.stdout:(32/33): librados2-19.3.0-5120.g18432218.el9.x8 12 MB/s | 3.4 MB 00:00 2024-09-21T01:18:01.579 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 30/35 2024-09-21T01:18:01.663 INFO:teuthology.orchestra.run.smithi090.stdout:(33/33): librbd1-19.3.0-5120.g18432218.el9.x86_ 10 MB/s | 3.2 MB 00:00 2024-09-21T01:18:01.668 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:01.669 INFO:teuthology.orchestra.run.smithi090.stdout:Total 17 MB/s | 59 MB 00:03 2024-09-21T01:18:01.783 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:18:01.820 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:18:01.820 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:18:02.224 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 30/35 2024-09-21T01:18:02.266 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 31/35 2024-09-21T01:18:02.344 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 31/35 2024-09-21T01:18:02.344 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T01:18:02.344 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T01:18:02.344 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:02.376 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 32/35 2024-09-21T01:18:02.520 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:18:02.521 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:18:04.010 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:18:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T01:18:04.143 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-21T01:18:04.177 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-21T01:18:04.208 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T01:18:04.235 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-21T01:18:04.820 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T01:18:04.886 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-21T01:18:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-21T01:18:05.406 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:19.3.0-5120.g18432218.el9.x86_64 7/35 2024-09-21T01:18:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:19.3.0-5120.g18432218.el9.x86_64 7/35 2024-09-21T01:18:05.705 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 8/35 2024-09-21T01:18:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/35 2024-09-21T01:18:05.796 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 9/35 2024-09-21T01:18:05.833 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-21T01:18:05.877 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:19.3.0-5120.g18432218.el9. 11/35 2024-09-21T01:18:05.949 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 12/35 2024-09-21T01:18:05.977 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 13/35 2024-09-21T01:18:06.042 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 13/35 2024-09-21T01:18:06.076 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-21T01:18:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-21T01:18:06.229 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-21T01:18:06.270 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-21T01:18:06.302 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/35 2024-09-21T01:18:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-21T01:18:06.355 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T01:18:06.394 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-21T01:18:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T01:18:06.779 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 23/35 2024-09-21T01:18:06.865 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 23/35 2024-09-21T01:18:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 24/35 2024-09-21T01:18:06.937 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T01:18:07.434 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T01:18:07.475 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 27/35 2024-09-21T01:18:07.546 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 27/35 2024-09-21T01:18:07.619 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 28/35 2024-09-21T01:18:07.652 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:19.3.0-5120.g18432218.el9.x8 29/35 2024-09-21T01:18:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 30/35 2024-09-21T01:18:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 30/35 2024-09-21T01:18:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 30/35 2024-09-21T01:18:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 31/35 2024-09-21T01:18:10.774 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 31/35 2024-09-21T01:18:10.774 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T01:18:10.774 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T01:18:10.774 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 32/35 2024-09-21T01:18:12.883 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 32/35 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /root 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-09-21T01:18:12.884 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:13.932 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 33/35 2024-09-21T01:18:13.959 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 33/35 2024-09-21T01:18:13.959 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:13.959 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T01:18:13.959 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:18:13.960 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:18:13.960 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:13.961 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T01:18:14.051 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T01:18:14.051 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T01:18:15.331 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T01:18:15.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 1/35 2024-09-21T01:18:15.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2/35 2024-09-21T01:18:15.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 3/35 2024-09-21T01:18:15.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 4/35 2024-09-21T01:18:15.331 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 5/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 6/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 7/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el9. 8/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9.x8 9/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 10/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 11/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 12/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 13/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T01:18:15.332 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 32/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T01:18:15.333 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 34/35 2024-09-21T01:18:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T01:18:16.398 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T01:18:16.399 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.400 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:16.401 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:16.814 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-09-21T01:18:17.426 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:29 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:17.563 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:17.563 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T01:18:17.564 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:17.565 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:18:17.565 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:17.565 INFO:teuthology.orchestra.run.smithi045.stdout:Install 4 Packages 2024-09-21T01:18:17.565 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:17.565 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 40 M 2024-09-21T01:18:17.565 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 197 M 2024-09-21T01:18:17.566 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:18:17.969 INFO:teuthology.orchestra.run.smithi045.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-21T01:18:18.010 INFO:teuthology.orchestra.run.smithi045.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-21T01:18:18.052 INFO:teuthology.orchestra.run.smithi045.stdout:(3/4): libcephsqlite-19.3.0-5120.g18432218.el9. 688 kB/s | 158 kB 00:00 2024-09-21T01:18:19.086 INFO:teuthology.orchestra.run.smithi045.stdout:(4/4): ceph-test-19.3.0-5120.g18432218.el9.x86_ 32 MB/s | 40 MB 00:01 2024-09-21T01:18:19.088 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:19.088 INFO:teuthology.orchestra.run.smithi045.stdout:Total 27 MB/s | 40 MB 00:01 2024-09-21T01:18:19.116 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:18:19.139 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:18:19.139 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:18:19.477 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:18:19.477 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:18:19.965 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:18:20.073 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T01:18:20.119 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T01:18:20.146 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 3/4 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 32/35 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-09-21T01:18:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 33/35 2024-09-21T01:18:22.734 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 33/35 2024-09-21T01:18:22.735 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:22.735 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T01:18:22.735 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:18:22.735 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T01:18:22.735 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:22.737 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T01:18:22.828 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T01:18:22.828 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 1/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 3/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 4/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 5/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_6 6/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:19.3.0-5120.g18432218.el9.x86_64 7/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5120.g18432218.el9. 8/35 2024-09-21T01:18:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:19.3.0-5120.g18432218.el9.x8 9/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 10/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 11/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 12/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 13/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T01:18:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:19.3.0-5120.g18432218.el9.x86_64 32/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T01:18:24.310 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:19.3.0-5120.g18432218.el9.x86_64 34/35 2024-09-21T01:18:24.426 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 3/4 2024-09-21T01:18:24.515 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 4/4 2024-09-21T01:18:25.200 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T01:18:25.200 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:25.200 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-09-21T01:18:25.200 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.200 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.200 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T01:18:25.201 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.202 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:25.203 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:25.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-09-21T01:18:26.188 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:31 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:26.328 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:19.3.0-5120.g18432218.el9 ceph 40 M 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:19.3.0-5120.g18432218.el9 ceph 158 k 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:26.329 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-09-21T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 40 M 2024-09-21T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 197 M 2024-09-21T01:18:26.330 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:18:26.594 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-21T01:18:26.644 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): libcephsqlite-19.3.0-5120.g18432218.el9. 802 kB/s | 158 kB 00:00 2024-09-21T01:18:26.669 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 850 kB/s | 64 kB 00:00 2024-09-21T01:18:26.846 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 4/4 2024-09-21T01:18:26.846 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 1/4 2024-09-21T01:18:26.846 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2/4 2024-09-21T01:18:26.846 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T01:18:27.133 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:27.134 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:27.406 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-09-21T01:18:27.803 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): ceph-test-19.3.0-5120.g18432218.el9.x86_ 29 MB/s | 40 MB 00:01 2024-09-21T01:18:27.805 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:27.806 INFO:teuthology.orchestra.run.smithi090.stdout:Total 27 MB/s | 40 MB 00:01 2024-09-21T01:18:27.833 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:18:27.854 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:18:27.855 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:18:28.009 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:40 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:28.155 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:28.157 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T01:18:28.158 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T01:18:28.159 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T01:18:28.160 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout:Install 37 Packages 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 31 M 2024-09-21T01:18:28.161 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 119 M 2024-09-21T01:18:28.162 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:18:28.173 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:18:28.174 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:18:28.669 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:18:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T01:18:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T01:18:28.848 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 3/4 2024-09-21T01:18:29.048 INFO:teuthology.orchestra.run.smithi045.stdout:(1/37): ceph-19.3.0-5120.g18432218.el9.x86_64.r 56 kB/s | 6.6 kB 00:00 2024-09-21T01:18:29.257 INFO:teuthology.orchestra.run.smithi045.stdout:(2/37): ceph-mds-19.3.0-5120.g18432218.el9.x86_ 7.0 MB/s | 2.3 MB 00:00 2024-09-21T01:18:29.299 INFO:teuthology.orchestra.run.smithi045.stdout:(3/37): ceph-mgr-19.3.0-5120.g18432218.el9.x86_ 4.1 MB/s | 1.5 MB 00:00 2024-09-21T01:18:29.399 INFO:teuthology.orchestra.run.smithi045.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5120.g1843 2.9 MB/s | 292 kB 00:00 2024-09-21T01:18:29.474 INFO:teuthology.orchestra.run.smithi045.stdout:(5/37): ceph-mon-19.3.0-5120.g18432218.el9.x86_ 12 MB/s | 5.0 MB 00:00 2024-09-21T01:18:29.591 INFO:teuthology.orchestra.run.smithi045.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-21T01:18:29.633 INFO:teuthology.orchestra.run.smithi045.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T01:18:29.675 INFO:teuthology.orchestra.run.smithi045.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-09-21T01:18:29.700 INFO:teuthology.orchestra.run.smithi045.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-21T01:18:29.742 INFO:teuthology.orchestra.run.smithi045.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-21T01:18:29.767 INFO:teuthology.orchestra.run.smithi045.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-21T01:18:29.960 INFO:teuthology.orchestra.run.smithi045.stdout:(12/37): ceph-osd-19.3.0-5120.g18432218.el9.x86 25 MB/s | 18 MB 00:00 2024-09-21T01:18:30.255 INFO:teuthology.orchestra.run.smithi045.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 118 kB/s | 35 kB 00:00 2024-09-21T01:18:30.414 INFO:teuthology.orchestra.run.smithi045.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 266 kB/s | 172 kB 00:00 2024-09-21T01:18:30.439 INFO:teuthology.orchestra.run.smithi045.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 271 kB/s | 188 kB 00:00 2024-09-21T01:18:30.531 INFO:teuthology.orchestra.run.smithi045.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 436 kB/s | 51 kB 00:00 2024-09-21T01:18:30.556 INFO:teuthology.orchestra.run.smithi045.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 256 kB/s | 77 kB 00:00 2024-09-21T01:18:30.573 INFO:teuthology.orchestra.run.smithi045.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 312 kB/s | 42 kB 00:00 2024-09-21T01:18:30.615 INFO:teuthology.orchestra.run.smithi045.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-21T01:18:30.632 INFO:teuthology.orchestra.run.smithi045.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 742 kB/s | 43 kB 00:00 2024-09-21T01:18:30.657 INFO:teuthology.orchestra.run.smithi045.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.2 MB/s | 172 kB 00:00 2024-09-21T01:18:30.683 INFO:teuthology.orchestra.run.smithi045.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-09-21T01:18:30.708 INFO:teuthology.orchestra.run.smithi045.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-21T01:18:30.733 INFO:teuthology.orchestra.run.smithi045.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 932 kB/s | 23 kB 00:00 2024-09-21T01:18:30.759 INFO:teuthology.orchestra.run.smithi045.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-21T01:18:30.784 INFO:teuthology.orchestra.run.smithi045.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-21T01:18:30.843 INFO:teuthology.orchestra.run.smithi045.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2024-09-21T01:18:30.868 INFO:teuthology.orchestra.run.smithi045.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 616 kB/s | 46 kB 00:00 2024-09-21T01:18:30.893 INFO:teuthology.orchestra.run.smithi045.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T01:18:30.919 INFO:teuthology.orchestra.run.smithi045.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 655 kB/s | 16 kB 00:00 2024-09-21T01:18:30.944 INFO:teuthology.orchestra.run.smithi045.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.6 MB/s | 90 kB 00:00 2024-09-21T01:18:30.969 INFO:teuthology.orchestra.run.smithi045.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-09-21T01:18:30.994 INFO:teuthology.orchestra.run.smithi045.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-21T01:18:31.028 INFO:teuthology.orchestra.run.smithi045.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-09-21T01:18:31.054 INFO:teuthology.orchestra.run.smithi045.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-21T01:18:31.079 INFO:teuthology.orchestra.run.smithi045.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-21T01:18:31.112 INFO:teuthology.orchestra.run.smithi045.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-09-21T01:18:31.117 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:31.117 INFO:teuthology.orchestra.run.smithi045.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-21T01:18:31.275 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:18:31.321 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:18:31.321 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:18:31.855 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:18:31.855 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:18:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 3/4 2024-09-21T01:18:33.423 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 4/4 2024-09-21T01:18:33.424 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:18:33.424 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T01:18:33.424 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T01:18:33.424 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T01:18:33.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T01:18:33.527 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T01:18:33.559 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T01:18:33.657 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T01:18:33.741 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T01:18:33.787 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T01:18:33.823 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T01:18:33.851 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T01:18:33.878 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T01:18:33.912 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T01:18:33.952 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T01:18:33.996 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T01:18:34.030 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T01:18:34.080 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T01:18:34.118 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T01:18:34.195 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T01:18:34.355 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T01:18:34.446 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T01:18:34.505 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T01:18:34.542 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T01:18:34.601 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T01:18:34.665 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T01:18:34.863 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T01:18:34.914 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T01:18:34.995 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T01:18:35.094 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T01:18:35.129 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T01:18:35.265 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T01:18:35.448 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9. 32/37 2024-09-21T01:18:35.480 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 33/37 2024-09-21T01:18:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 33/37 2024-09-21T01:18:35.509 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:35.509 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T01:18:35.509 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:18:35.509 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:18:35.509 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 4/4 2024-09-21T01:18:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 1/4 2024-09-21T01:18:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2/4 2024-09-21T01:18:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:36.411 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:36.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-09-21T01:18:37.156 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 34/37 2024-09-21T01:18:37.185 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 34/37 2024-09-21T01:18:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:37.185 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T01:18:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:18:37.186 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:18:37.186 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:37.326 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:42 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:37.465 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:18:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:19.3.0-5120.g18432218.el9 ceph 6.6 k 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:19.3.0-5120.g18432218.el9 ceph 2.3 M 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:19.3.0-5120.g18432218.el9 ceph 1.5 M 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 292 k 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:19.3.0-5120.g18432218.el9 ceph 5.0 M 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:19.3.0-5120.g18432218.el9 ceph 18 M 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T01:18:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T01:18:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T01:18:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout:Install 37 Packages 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:37.471 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 31 M 2024-09-21T01:18:37.472 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 119 M 2024-09-21T01:18:37.472 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:18:37.682 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 35/37 2024-09-21T01:18:37.707 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 35/37 2024-09-21T01:18:37.707 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:37.707 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T01:18:37.707 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:18:37.707 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:18:37.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:37.785 INFO:teuthology.orchestra.run.smithi090.stdout:(1/37): ceph-19.3.0-5120.g18432218.el9.x86_64.r 57 kB/s | 6.6 kB 00:00 2024-09-21T01:18:37.950 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 36/37 2024-09-21T01:18:37.975 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 36/37 2024-09-21T01:18:37.975 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:37.975 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T01:18:37.976 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:18:37.976 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:18:37.976 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:38.003 INFO:teuthology.orchestra.run.smithi090.stdout:(2/37): ceph-mds-19.3.0-5120.g18432218.el9.x86_ 6.9 MB/s | 2.3 MB 00:00 2024-09-21T01:18:38.042 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 37/37 2024-09-21T01:18:38.045 INFO:teuthology.orchestra.run.smithi090.stdout:(3/37): ceph-mgr-19.3.0-5120.g18432218.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-09-21T01:18:38.154 INFO:teuthology.orchestra.run.smithi090.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5120.g1843 2.6 MB/s | 292 kB 00:00 2024-09-21T01:18:38.263 INFO:teuthology.orchestra.run.smithi090.stdout:(5/37): ceph-mon-19.3.0-5120.g18432218.el9.x86_ 10 MB/s | 5.0 MB 00:00 2024-09-21T01:18:38.346 INFO:teuthology.orchestra.run.smithi090.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-21T01:18:38.388 INFO:teuthology.orchestra.run.smithi090.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T01:18:38.430 INFO:teuthology.orchestra.run.smithi090.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-21T01:18:38.639 INFO:teuthology.orchestra.run.smithi090.stdout:(9/37): ceph-osd-19.3.0-5120.g18432218.el9.x86_ 28 MB/s | 18 MB 00:00 2024-09-21T01:18:38.664 INFO:teuthology.orchestra.run.smithi090.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 534 kB/s | 125 kB 00:00 2024-09-21T01:18:38.715 INFO:teuthology.orchestra.run.smithi090.stdout:(11/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-21T01:18:38.782 INFO:teuthology.orchestra.run.smithi090.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.5 MB/s | 215 kB 00:00 2024-09-21T01:18:38.907 INFO:teuthology.orchestra.run.smithi090.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 777 kB/s | 188 kB 00:00 2024-09-21T01:18:38.932 INFO:teuthology.orchestra.run.smithi090.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 231 kB/s | 35 kB 00:00 2024-09-21T01:18:38.966 INFO:teuthology.orchestra.run.smithi090.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 686 kB/s | 172 kB 00:00 2024-09-21T01:18:38.991 INFO:teuthology.orchestra.run.smithi090.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 918 kB/s | 77 kB 00:00 2024-09-21T01:18:39.017 INFO:teuthology.orchestra.run.smithi090.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 826 kB/s | 42 kB 00:00 2024-09-21T01:18:39.050 INFO:teuthology.orchestra.run.smithi090.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 433 kB/s | 51 kB 00:00 2024-09-21T01:18:39.092 INFO:teuthology.orchestra.run.smithi090.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 221 kB/s | 22 kB 00:00 2024-09-21T01:18:39.126 INFO:teuthology.orchestra.run.smithi090.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 576 kB/s | 43 kB 00:00 2024-09-21T01:18:39.151 INFO:teuthology.orchestra.run.smithi090.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-09-21T01:18:39.177 INFO:teuthology.orchestra.run.smithi090.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 428 kB/s | 11 kB 00:00 2024-09-21T01:18:39.202 INFO:teuthology.orchestra.run.smithi090.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 703 kB/s | 18 kB 00:00 2024-09-21T01:18:39.236 INFO:teuthology.orchestra.run.smithi090.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-21T01:18:39.261 INFO:teuthology.orchestra.run.smithi090.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-09-21T01:18:39.288 INFO:teuthology.orchestra.run.smithi090.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 377 kB/s | 19 kB 00:00 2024-09-21T01:18:39.313 INFO:teuthology.orchestra.run.smithi090.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 381 kB/s | 20 kB 00:00 2024-09-21T01:18:39.338 INFO:teuthology.orchestra.run.smithi090.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-21T01:18:39.364 INFO:teuthology.orchestra.run.smithi090.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-21T01:18:39.389 INFO:teuthology.orchestra.run.smithi090.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T01:18:39.415 INFO:teuthology.orchestra.run.smithi090.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 325 kB/s | 16 kB 00:00 2024-09-21T01:18:39.440 INFO:teuthology.orchestra.run.smithi090.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-21T01:18:39.465 INFO:teuthology.orchestra.run.smithi090.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-21T01:18:39.499 INFO:teuthology.orchestra.run.smithi090.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T01:18:39.533 INFO:teuthology.orchestra.run.smithi090.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 601 kB/s | 20 kB 00:00 2024-09-21T01:18:39.558 INFO:teuthology.orchestra.run.smithi090.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-09-21T01:18:39.583 INFO:teuthology.orchestra.run.smithi090.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-21T01:18:39.586 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:39.586 INFO:teuthology.orchestra.run.smithi090.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-21T01:18:39.733 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:18:39.773 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:18:39.773 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:19.3.0-5120.g18432218.el9.x86_64 37/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 3/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 4/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 5/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9. 6/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T01:18:39.833 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T01:18:39.834 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T01:18:39.835 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T01:18:39.835 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T01:18:39.836 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T01:18:40.261 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:18:40.261 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:18:40.770 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T01:18:40.770 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:40.770 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T01:18:40.771 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T01:18:40.772 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:40.773 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:41.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-09-21T01:18:41.671 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:18:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T01:18:41.781 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T01:18:41.795 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:41.814 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T01:18:41.887 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:18:41.894 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T01:18:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T01:18:41.932 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:41.933 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:18:41.933 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T01:18:42.003 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-09-21T01:18:42.065 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T01:18:42.144 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T01:18:42.194 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T01:18:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T01:18:42.266 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T01:18:42.293 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T01:18:42.326 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T01:18:42.368 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T01:18:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T01:18:42.446 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T01:18:42.495 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T01:18:42.535 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T01:18:42.602 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:42.608 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T01:18:42.739 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:42.740 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:18:42.741 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:42.741 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 784 k 2024-09-21T01:18:42.741 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 791 k 2024-09-21T01:18:42.741 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:18:42.755 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T01:18:42.835 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T01:18:42.890 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T01:18:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T01:18:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T01:18:43.025 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-19.3.0-5120.g18432218.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-21T01:18:43.025 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:43.026 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-21T01:18:43.026 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:18:43.031 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:18:43.032 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:18:43.038 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:18:43.038 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:18:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T01:18:43.111 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:18:43.252 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T01:18:43.308 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T01:18:43.386 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T01:18:43.475 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T01:18:43.517 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T01:18:43.536 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:43.633 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:43.645 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T01:18:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9. 32/37 2024-09-21T01:18:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 33/37 2024-09-21T01:18:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 33/37 2024-09-21T01:18:43.863 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:43.863 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T01:18:43.864 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:18:43.864 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T01:18:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:44.009 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:44.249 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:44.249 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:44.249 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:18:44.249 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:44.249 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:44.249 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:44.424 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T01:18:45.025 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:45.164 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:45.164 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:18:45.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:45.166 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 148 k 2024-09-21T01:18:45.166 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 486 k 2024-09-21T01:18:45.166 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:18:45.349 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-19.3.0-5120.g184322 811 kB/s | 148 kB 00:00 2024-09-21T01:18:45.349 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:45.349 INFO:teuthology.orchestra.run.smithi045.stdout:Total 804 kB/s | 148 kB 00:00 2024-09-21T01:18:45.350 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:18:45.358 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:18:45.358 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:18:45.425 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:18:45.426 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:18:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 34/37 2024-09-21T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 34/37 2024-09-21T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T01:18:45.458 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:45.602 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:18:45.667 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432218. 1/1 2024-09-21T01:18:45.693 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432218. 1/1 2024-09-21T01:18:45.693 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:45.693 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T01:18:45.693 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:46.005 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 35/37 2024-09-21T01:18:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 35/37 2024-09-21T01:18:46.030 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:46.030 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T01:18:46.030 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:18:46.030 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T01:18:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:46.256 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 36/37 2024-09-21T01:18:46.280 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 36/37 2024-09-21T01:18:46.281 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:46.281 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T01:18:46.281 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:18:46.281 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T01:18:46.281 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:19.3.0-5120.g18432218.el9.x86_64 37/37 2024-09-21T01:18:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432218. 1/1 2024-09-21T01:18:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:46.637 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:18:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:46.637 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:46.637 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:46.845 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-09-21T01:18:47.450 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:47.543 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:18:47.587 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:47.588 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:18:47.588 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:47.661 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T01:18:48.267 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:48.405 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:48.406 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T01:18:48.407 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:48.408 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 19 M 2024-09-21T01:18:48.409 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 159 M 2024-09-21T01:18:48.409 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:18:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:19.3.0-5120.g18432218.el9.x86_64 37/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:19.3.0-5120.g18432218.el9.x86_64 1/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 3/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 4/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 5/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9. 6/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T01:18:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T01:18:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T01:18:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T01:18:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T01:18:49.054 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): ceph-grafana-dashboards-19.3.0-5120.g18 214 kB/s | 34 kB 00:00 2024-09-21T01:18:49.071 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): ceph-prometheus-alerts-19.3.0-5120.g184 93 kB/s | 16 kB 00:00 2024-09-21T01:18:49.271 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-21T01:18:49.338 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-09-21T01:18:49.388 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 7.2 MB/s | 863 kB 00:00 2024-09-21T01:18:49.438 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.3 MB/s | 548 kB 00:00 2024-09-21T01:18:49.463 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 258 kB/s | 19 kB 00:00 2024-09-21T01:18:49.497 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.2 MB/s | 144 kB 00:00 2024-09-21T01:18:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T01:18:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T01:18:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T01:18:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T01:18:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T01:18:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T01:18:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:49.535 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:49.656 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): ceph-mgr-dashboard-19.3.0-5120.g1843221 19 MB/s | 14 MB 00:00 2024-09-21T01:18:49.681 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 167 kB/s | 31 kB 00:00 2024-09-21T01:18:49.748 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.6 MB/s | 2.0 MB 00:00 2024-09-21T01:18:49.781 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-09-21T01:18:49.787 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:49.787 INFO:teuthology.orchestra.run.smithi045.stdout:Total 14 MB/s | 19 MB 00:01 2024-09-21T01:18:49.839 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:18:49.861 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:18:49.861 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:18:49.956 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-09-21T01:18:50.142 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:18:50.143 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:18:50.534 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:50.623 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:18:50.665 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:50.666 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:18:50.666 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:50.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-09-21T01:18:50.754 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:18:50.841 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T01:18:50.877 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T01:18:50.924 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T01:18:50.953 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T01:18:51.047 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T01:18:51.177 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T01:18:51.269 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T01:18:51.318 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T01:18:51.323 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:51.345 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T01:18:51.378 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9 10/12 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 784 k 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:51.453 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:18:51.454 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:51.454 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:18:51.454 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:51.454 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 784 k 2024-09-21T01:18:51.454 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 791 k 2024-09-21T01:18:51.455 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:18:51.756 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-19.3.0-5120.g18432218.el9.noarch.rpm 2.5 MB/s | 784 kB 00:00 2024-09-21T01:18:51.756 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:51.757 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.5 MB/s | 784 kB 00:00 2024-09-21T01:18:51.757 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:18:51.762 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:18:51.762 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:18:51.768 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:18:51.768 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:18:51.841 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:18:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:52.386 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:53.077 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:18:53.077 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:53.077 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:18:53.077 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:53.077 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:53.077 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:53.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T01:18:53.914 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:58 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:54.044 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5120.g18432218.el9 ceph 148 k 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:54.045 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:18:54.046 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:54.046 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 148 k 2024-09-21T01:18:54.046 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 486 k 2024-09-21T01:18:54.046 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:18:54.252 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-19.3.0-5120.g184322 721 kB/s | 148 kB 00:00 2024-09-21T01:18:54.253 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:54.253 INFO:teuthology.orchestra.run.smithi090.stdout:Total 714 kB/s | 148 kB 00:00 2024-09-21T01:18:54.253 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:18:54.261 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:18:54.261 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:18:54.323 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:18:54.323 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:18:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:18:54.579 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5120.g18432218. 1/1 2024-09-21T01:18:54.604 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5120.g18432218. 1/1 2024-09-21T01:18:54.604 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:18:54.604 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T01:18:54.604 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:54.764 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el 11/12 2024-09-21T01:18:54.823 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noa 12/12 2024-09-21T01:18:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5120.g18432218. 1/1 2024-09-21T01:18:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:55.604 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:18:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:18:55.604 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:55.604 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:55.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-09-21T01:18:56.489 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:01 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:56.580 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:18:56.623 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:56.624 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:18:56.624 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:18:56.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T01:18:56.852 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noa 12/12 2024-09-21T01:18:56.852 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el 1/12 2024-09-21T01:18:56.852 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noa 2/12 2024-09-21T01:18:56.852 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9 3/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T01:18:56.853 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T01:18:57.173 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T01:18:57.174 INFO:teuthology.orchestra.run.smithi045.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T01:18:57.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T01:18:57.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T01:18:57.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T01:18:57.175 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:57.175 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:18:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:02 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:18:57.395 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 14 M 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T01:18:57.396 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 34 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 16 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-09-21T01:18:57.397 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:18:57.398 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 19 M 2024-09-21T01:18:57.398 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 159 M 2024-09-21T01:18:57.398 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:18:57.417 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T01:18:58.019 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): ceph-grafana-dashboards-19.3.0-5120.g18 212 kB/s | 34 kB 00:00 2024-09-21T01:18:58.031 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:10 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:18:58.045 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): ceph-prometheus-alerts-19.3.0-5120.g184 88 kB/s | 16 kB 00:00 2024-09-21T01:18:58.173 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T01:18:58.174 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout:======================================================================================= 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout:Install 12 Packages 2024-09-21T01:18:58.175 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:18:58.176 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 38 M 2024-09-21T01:18:58.176 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 212 M 2024-09-21T01:18:58.176 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:18:58.571 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): ceph-mgr-dashboard-19.3.0-5120.g1843221 20 MB/s | 14 MB 00:00 2024-09-21T01:18:58.679 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 425 kB/s | 269 kB 00:00 2024-09-21T01:18:58.691 INFO:teuthology.orchestra.run.smithi045.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 804 kB/s | 191 kB 00:00 2024-09-21T01:18:58.729 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-21T01:18:58.755 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 774 kB/s | 19 kB 00:00 2024-09-21T01:18:58.758 INFO:teuthology.orchestra.run.smithi045.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-09-21T01:18:58.789 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-09-21T01:18:58.816 INFO:teuthology.orchestra.run.smithi045.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-09-21T01:18:58.864 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-09-21T01:18:58.889 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T01:18:58.915 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-09-21T01:18:58.933 INFO:teuthology.orchestra.run.smithi045.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-51 15 MB/s | 7.4 MB 00:00 2024-09-21T01:18:58.940 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-21T01:18:58.967 INFO:teuthology.orchestra.run.smithi045.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 121 kB/s | 18 kB 00:00 2024-09-21T01:18:59.034 INFO:teuthology.orchestra.run.smithi045.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-21T01:18:59.048 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:01 2024-09-21T01:18:59.052 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:59.053 INFO:teuthology.orchestra.run.smithi090.stdout:Total 12 MB/s | 19 MB 00:01 2024-09-21T01:18:59.059 INFO:teuthology.orchestra.run.smithi045.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 306 kB/s | 38 kB 00:00 2024-09-21T01:18:59.101 INFO:teuthology.orchestra.run.smithi045.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-09-21T01:18:59.102 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:18:59.121 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:18:59.121 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:18:59.143 INFO:teuthology.orchestra.run.smithi045.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-09-21T01:18:59.269 INFO:teuthology.orchestra.run.smithi045.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 16 MB/s | 4.9 MB 00:00 2024-09-21T01:18:59.386 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:18:59.386 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:18:59.536 INFO:teuthology.orchestra.run.smithi045.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-09-21T01:18:59.602 INFO:teuthology.orchestra.run.smithi045.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.4 MB/s | 5.1 MB 00:00 2024-09-21T01:18:59.603 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:18:59.603 INFO:teuthology.orchestra.run.smithi045.stdout:Total 27 MB/s | 38 MB 00:01 2024-09-21T01:18:59.968 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:18:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:18:59.995 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:18:59.995 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:00.058 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T01:19:00.102 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T01:19:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T01:19:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T01:19:00.286 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T01:19:00.422 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T01:19:00.457 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:00.457 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:00.518 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T01:19:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T01:19:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T01:19:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9 10/12 2024-09-21T01:19:01.354 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:01.421 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T01:19:01.467 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T01:19:01.573 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T01:19:01.611 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T01:19:01.802 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T01:19:01.834 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T01:19:01.922 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T01:19:02.618 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T01:19:02.736 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T01:19:03.803 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el 11/12 2024-09-21T01:19:04.105 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T01:19:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noa 12/12 2024-09-21T01:19:05.131 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T01:19:05.152 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432 12/12 2024-09-21T01:19:06.467 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noa 12/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el 1/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noa 2/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9 3/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T01:19:06.468 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T01:19:06.780 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:06.781 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:07.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T01:19:07.615 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:12 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432 12/12 2024-09-21T01:19:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432 1/12 2024-09-21T01:19:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T01:19:07.697 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T01:19:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T01:19:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T01:19:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T01:19:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T01:19:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T01:19:07.698 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T01:19:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T01:19:07.699 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T01:19:07.746 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 7.4 M 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:19:07.747 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout:======================================================================================= 2024-09-21T01:19:07.748 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-09-21T01:19:07.749 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:07.749 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 38 M 2024-09-21T01:19:07.749 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 212 M 2024-09-21T01:19:07.749 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T01:19:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:08.173 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:08.173 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T01:19:08.174 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T01:19:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T01:19:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T01:19:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T01:19:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T01:19:08.175 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:08.175 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:08.425 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-51 15 MB/s | 7.4 MB 00:00 2024-09-21T01:19:08.550 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 33 kB 00:00 2024-09-21T01:19:08.566 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2024-09-21T01:19:08.751 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-09-21T01:19:08.843 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 199 kB/s | 18 kB 00:00 2024-09-21T01:19:08.876 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): libquadmath-11.5.0-2.el9.x86_64.rpm 201 kB/s | 191 kB 00:00 2024-09-21T01:19:08.926 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 460 kB/s | 38 kB 00:00 2024-09-21T01:19:09.010 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-21T01:19:09.052 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): libgfortran-11.5.0-2.el9.x86_64.rpm 711 kB/s | 801 kB 00:01 2024-09-21T01:19:09.194 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-09-21T01:19:09.203 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:21 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:09.287 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 12 MB/s | 4.9 MB 00:00 2024-09-21T01:19:09.345 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 858 kB/s | 250 kB 00:00 2024-09-21T01:19:09.345 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:09.346 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T01:19:09.347 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:09.348 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.2 M 2024-09-21T01:19:09.349 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 26 M 2024-09-21T01:19:09.349 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:09.695 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-21T01:19:09.696 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:09.697 INFO:teuthology.orchestra.run.smithi090.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-21T01:19:09.970 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): ceph-mgr-rook-19.3.0-5120.g18432218.el9 310 kB/s | 49 kB 00:00 2024-09-21T01:19:10.055 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:10.081 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:10.081 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:10.096 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 67 kB/s | 19 kB 00:00 2024-09-21T01:19:10.121 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 83 kB/s | 26 kB 00:00 2024-09-21T01:19:10.515 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:10.516 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:10.563 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 340 kB/s | 159 kB 00:00 2024-09-21T01:19:10.588 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 597 kB/s | 279 kB 00:00 2024-09-21T01:19:10.622 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 341 kB/s | 222 kB 00:00 2024-09-21T01:19:10.664 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-09-21T01:19:10.689 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 54 kB 00:00 2024-09-21T01:19:10.714 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-21T01:19:10.748 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T01:19:10.782 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-21T01:19:10.807 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.6 MB/s | 90 kB 00:00 2024-09-21T01:19:10.849 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-21T01:19:10.853 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:10.853 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-09-21T01:19:10.926 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:10.943 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:10.943 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:11.042 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:11.044 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:11.276 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:11.353 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T01:19:11.400 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T01:19:11.502 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T01:19:11.538 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T01:19:11.568 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:11.721 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T01:19:11.745 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T01:19:11.748 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T01:19:11.811 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T01:19:11.821 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T01:19:11.867 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T01:19:11.911 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T01:19:11.953 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T01:19:12.060 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T01:19:12.186 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T01:19:12.247 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T01:19:12.464 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T01:19:12.604 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T01:19:13.034 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T01:19:13.077 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T01:19:13.110 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T01:19:13.160 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T01:19:13.188 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 13/13 2024-09-21T01:19:13.868 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 13/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 1/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T01:19:13.918 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T01:19:13.919 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T01:19:13.919 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T01:19:13.919 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T01:19:13.919 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T01:19:13.919 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T01:19:14.254 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T01:19:14.254 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:14.254 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T01:19:14.255 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T01:19:14.256 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:14.256 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:14.509 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T01:19:14.912 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T01:19:14.946 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432 12/12 2024-09-21T01:19:15.121 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:15.262 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T01:19:15.263 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 7.0 M 2024-09-21T01:19:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 33 M 2024-09-21T01:19:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:15.611 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-21T01:19:15.661 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T01:19:15.687 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): ceph-mgr-cephadm-19.3.0-5120.g18432218.e 697 kB/s | 154 kB 00:00 2024-09-21T01:19:15.704 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.3 MB/s | 58 kB 00:00 2024-09-21T01:19:15.738 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-21T01:19:15.821 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-21T01:19:15.823 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:15.824 INFO:teuthology.orchestra.run.smithi045.stdout:Total 13 MB/s | 7.0 MB 00:00 2024-09-21T01:19:15.896 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:15.908 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:15.910 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:16.016 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:16.016 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:16.332 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:16.494 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T01:19:16.549 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T01:19:17.102 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T01:19:17.188 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T01:19:17.263 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T01:19:17.294 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 6/6 2024-09-21T01:19:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 6/6 2024-09-21T01:19:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/6 2024-09-21T01:19:18.031 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T01:19:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T01:19:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T01:19:18.032 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T01:19:18.117 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432 12/12 2024-09-21T01:19:18.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432 1/12 2024-09-21T01:19:18.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T01:19:18.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T01:19:18.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T01:19:18.117 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T01:19:18.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T01:19:18.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T01:19:18.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T01:19:18.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T01:19:18.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T01:19:18.118 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T01:19:18.308 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T01:19:18.308 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:18.309 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T01:19:18.540 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:18.541 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:18.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-09-21T01:19:18.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-09-21T01:19:19.168 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:19.307 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:19.307 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:19.307 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:19:19.307 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:19.308 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-09-21T01:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 953 k 2024-09-21T01:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 3.1 M 2024-09-21T01:19:19.309 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:19.540 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 612 kB/s | 80 kB 00:00 2024-09-21T01:19:19.555 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:19.686 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 49 k 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T01:19:19.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-09-21T01:19:19.689 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:19.690 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.2 M 2024-09-21T01:19:19.690 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-09-21T01:19:19.690 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:19.699 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): ceph-fuse-19.3.0-5120.g18432218.el9.x86_ 3.0 MB/s | 873 kB 00:00 2024-09-21T01:19:19.699 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:19.699 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.4 MB/s | 953 kB 00:00 2024-09-21T01:19:19.711 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:19.735 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:19.735 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:19.820 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:19.821 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:20.015 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:20.158 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T01:19:20.177 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 196 kB/s | 19 kB 00:00 2024-09-21T01:19:20.202 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 209 kB/s | 26 kB 00:00 2024-09-21T01:19:20.235 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2/2 2024-09-21T01:19:20.285 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-21T01:19:20.311 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): ceph-mgr-rook-19.3.0-5120.g18432218.el9 213 kB/s | 49 kB 00:00 2024-09-21T01:19:20.337 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-21T01:19:20.379 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.9 MB/s | 279 kB 00:00 2024-09-21T01:19:20.413 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-09-21T01:19:20.449 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 395 kB/s | 54 kB 00:00 2024-09-21T01:19:20.480 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-21T01:19:20.506 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.5 MB/s | 220 kB 00:00 2024-09-21T01:19:20.531 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-21T01:19:20.556 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-21T01:19:20.598 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-21T01:19:20.601 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:20.601 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-21T01:19:20.667 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:20.682 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:20.682 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:20.770 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:20.770 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:20.959 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2/2 2024-09-21T01:19:20.959 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/2 2024-09-21T01:19:21.220 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T01:19:21.220 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:21.220 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:21.220 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T01:19:21.220 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:21.220 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:21.298 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:21.470 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T01:19:21.508 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-volume 2024-09-21T01:19:21.525 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T01:19:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T01:19:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T01:19:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T01:19:21.769 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T01:19:21.880 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T01:19:21.938 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T01:19:22.121 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:22.262 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:22.262 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:22.262 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-09-21T01:19:22.262 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:19:22.263 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:22.264 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 285 k 2024-09-21T01:19:22.264 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 1.3 M 2024-09-21T01:19:22.264 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:22.505 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-volume-19.3.0-5120.g18432218.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T01:19:22.506 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:22.506 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-21T01:19:22.506 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:22.516 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:22.516 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:22.544 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:22.545 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:22.641 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T01:19:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T01:19:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T01:19:22.779 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T01:19:22.824 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 13/13 2024-09-21T01:19:22.849 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:19:22.867 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:19:22.867 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:19:22.867 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T01:19:22.867 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:23.825 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 13/13 2024-09-21T01:19:23.825 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 1/13 2024-09-21T01:19:23.825 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T01:19:23.825 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T01:19:23.825 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T01:19:23.825 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T01:19:23.826 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T01:19:23.968 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:19:23.969 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:23.969 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:23.969 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:23.969 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:23.969 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T01:19:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T01:19:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T01:19:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T01:19:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T01:19:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T01:19:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:24.158 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:24.287 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-09-21T01:19:24.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T01:19:24.899 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:25.030 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:25.039 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:25.040 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:19:25.041 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:25.041 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 127 k 2024-09-21T01:19:25.041 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 457 k 2024-09-21T01:19:25.041 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:25.162 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:25.163 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:25.163 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:19:25.165 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:25.165 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 154 k 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-09-21T01:19:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:25.167 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 7.0 M 2024-09-21T01:19:25.167 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-09-21T01:19:25.167 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:25.237 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-19.3.0-5120.g18432218.el9.x86_64 649 kB/s | 127 kB 00:00 2024-09-21T01:19:25.237 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:25.238 INFO:teuthology.orchestra.run.smithi045.stdout:Total 643 kB/s | 127 kB 00:00 2024-09-21T01:19:25.238 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:25.244 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:25.244 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:25.312 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:25.312 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:25.471 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:25.571 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:25.775 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-21T01:19:25.808 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): ceph-mgr-cephadm-19.3.0-5120.g18432218.e 613 kB/s | 154 kB 00:00 2024-09-21T01:19:25.867 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-21T01:19:25.892 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 689 kB/s | 58 kB 00:00 2024-09-21T01:19:25.917 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-21T01:19:25.980 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:26.076 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-21T01:19:26.079 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:26.079 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.7 MB/s | 7.0 MB 00:00 2024-09-21T01:19:26.149 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:26.160 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:26.161 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:26.242 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:26.243 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:26.243 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:26.243 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:26.243 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:26.243 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:26.261 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:26.262 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:26.515 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-09-21T01:19:26.571 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:26.741 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T01:19:26.795 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T01:19:27.126 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:27.221 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:27.267 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:27.268 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:27.268 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T01:19:27.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-09-21T01:19:27.440 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T01:19:27.516 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T01:19:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 6/6 2024-09-21T01:19:27.977 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:28.121 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:28.122 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-09-21T01:19:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:28.122 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:28.122 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 32 k 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 142 k 2024-09-21T01:19:28.123 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:28.268 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-19.3.0-5120.g18432218.el9.x86_6 219 kB/s | 32 kB 00:00 2024-09-21T01:19:28.269 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:28.269 INFO:teuthology.orchestra.run.smithi045.stdout:Total 216 kB/s | 32 kB 00:00 2024-09-21T01:19:28.269 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:28.273 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:28.273 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:28.291 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:28.291 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:28.390 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:28.405 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 6/6 2024-09-21T01:19:28.405 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 1/6 2024-09-21T01:19:28.405 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T01:19:28.405 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T01:19:28.405 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T01:19:28.405 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T01:19:28.494 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T01:19:28.664 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T01:19:28.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:28.665 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:28.910 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-09-21T01:19:28.914 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:29.159 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:29.159 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:29.159 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:29.159 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:29.159 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:29.160 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:29.353 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-09-21T01:19:29.485 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:29.619 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 873 k 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:29.620 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:29.621 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:29.621 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-09-21T01:19:29.621 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:29.621 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 953 k 2024-09-21T01:19:29.621 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 3.1 M 2024-09-21T01:19:29.621 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:29.857 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 584 kB/s | 80 kB 00:00 2024-09-21T01:19:29.960 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:29.998 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-19.3.0-5120.g18432218.el9.x86_ 3.1 MB/s | 873 kB 00:00 2024-09-21T01:19:30.007 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:30.007 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.5 MB/s | 953 kB 00:00 2024-09-21T01:19:30.011 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:30.033 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:30.033 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:30.053 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:30.099 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:30.100 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:30.100 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:30.113 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:30.114 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:30.169 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-09-21T01:19:30.297 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:30.470 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T01:19:30.542 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2/2 2024-09-21T01:19:30.774 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:30.867 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:30.914 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:30.915 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:30.915 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:30.984 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-09-21T01:19:31.273 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2/2 2024-09-21T01:19:31.273 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/2 2024-09-21T01:19:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T01:19:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:31.535 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:19.3.0-5120.g18432218.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T01:19:31.535 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:31.535 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:31.592 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:31.686 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:31.734 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:31.735 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:31.735 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:31.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-09-21T01:19:31.881 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-volume 2024-09-21T01:19:32.407 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:32.471 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:32.502 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:32.550 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:32.551 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:32.551 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:32.613 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:32.614 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:32.614 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-21T01:19:32.614 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:32.614 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:32.614 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:19.3.0-5120.g18432218.el9 ceph-noarch 285 k 2024-09-21T01:19:32.614 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 285 k 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.3 M 2024-09-21T01:19:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:32.624 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-09-21T01:19:32.843 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-volume-19.3.0-5120.g18432218.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T01:19:32.843 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:32.843 INFO:teuthology.orchestra.run.smithi090.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T01:19:32.844 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:32.852 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:32.852 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:32.881 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:32.882 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:19:33.187 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:19:33.188 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:19:33.188 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T01:19:33.188 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:33.234 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:33.328 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:33.374 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:33.375 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:33.375 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:33.442 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-09-21T01:19:34.052 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:34.146 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:34.192 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-09-21T01:19:34.193 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:34.253 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 1/1 2024-09-21T01:19:34.253 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:34.253 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:34.253 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:19.3.0-5120.g18432218.el9.noarch 2024-09-21T01:19:34.253 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:34.253 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:34.268 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-09-21T01:19:34.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-09-21T01:19:34.873 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:35.011 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 86 k 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 260 k 2024-09-21T01:19:35.012 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:35.172 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:35.214 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-19.3.0-5120.g18432218.el9.x86_64.rpm 430 kB/s | 86 kB 00:00 2024-09-21T01:19:35.214 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:35.214 INFO:teuthology.orchestra.run.smithi045.stdout:Total 427 kB/s | 86 kB 00:00 2024-09-21T01:19:35.214 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:35.222 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:35.222 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:35.289 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:35.289 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:35.312 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:35.312 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:35.312 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-21T01:19:35.312 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:35.312 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 127 k 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:35.313 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 127 k 2024-09-21T01:19:35.314 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 457 k 2024-09-21T01:19:35.314 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:35.476 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:35.523 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-19.3.0-5120.g18432218.el9.x86_64 606 kB/s | 127 kB 00:00 2024-09-21T01:19:35.524 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:35.524 INFO:teuthology.orchestra.run.smithi090.stdout:Total 600 kB/s | 127 kB 00:00 2024-09-21T01:19:35.524 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:35.530 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:35.530 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:35.590 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:35.596 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:35.597 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:35.856 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:35.994 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:36.250 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:36.250 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:36.250 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:36.250 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:36.250 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:36.250 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:36.450 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:36.551 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-09-21T01:19:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:36.711 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:36.712 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:37.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-09-21T01:19:37.163 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:37.304 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.1 M 2024-09-21T01:19:37.305 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 15 M 2024-09-21T01:19:37.306 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:37.626 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:37.626 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-19.3.0-5120.g18432218.el9.x86_64.rpm 9.7 MB/s | 3.1 MB 00:00 2024-09-21T01:19:37.627 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:37.627 INFO:teuthology.orchestra.run.smithi045.stdout:Total 9.6 MB/s | 3.1 MB 00:00 2024-09-21T01:19:37.628 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:37.638 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:37.638 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:37.717 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:37.725 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:37.726 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:37.765 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:37.766 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:37.766 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:37.832 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-09-21T01:19:38.201 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:38.261 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:38.287 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:38.287 INFO:teuthology.orchestra.run.smithi045.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:19:38.287 INFO:teuthology.orchestra.run.smithi045.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T01:19:38.288 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:19:38.288 INFO:teuthology.orchestra.run.smithi045.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:19:38.288 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:38.414 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:38.548 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:19.3.0-5120.g18432218.el9 ceph 32 k 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 32 k 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 142 k 2024-09-21T01:19:38.549 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:38.873 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-19.3.0-5120.g18432218.el9.x86_6 98 kB/s | 32 kB 00:00 2024-09-21T01:19:38.873 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:38.873 INFO:teuthology.orchestra.run.smithi090.stdout:Total 98 kB/s | 32 kB 00:00 2024-09-21T01:19:38.873 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:38.877 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:38.896 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:38.896 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:38.988 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:39.516 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:39.517 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:39.517 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:39.517 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:39.517 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:39.517 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:39.763 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:39.764 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:39.764 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:39.764 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:39.764 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:39.764 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:39.767 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-09-21T01:19:39.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-09-21T01:19:40.375 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:52 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:19:40.516 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:19:40.516 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 170 k 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 545 k 2024-09-21T01:19:40.517 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:19:40.542 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:40.630 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:40.673 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:40.675 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:40.675 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:40.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-09-21T01:19:40.865 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-19.3.0-5120.g18432218.el9.x86_64.rpm 488 kB/s | 170 kB 00:00 2024-09-21T01:19:40.866 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:40.866 INFO:teuthology.orchestra.run.smithi045.stdout:Total 486 kB/s | 170 kB 00:00 2024-09-21T01:19:40.866 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:19:40.875 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:19:40.875 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:19:40.945 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:19:40.945 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:19:41.108 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:19:41.207 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:41.374 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:41.463 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:41.506 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:41.507 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:41.507 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:41.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-09-21T01:19:41.636 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:41.898 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:41.899 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:41.899 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:19:41.899 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:41.899 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:19:41.899 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:19:42.163 DEBUG:teuthology.parallel:result is None 2024-09-21T01:19:42.176 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:42.266 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:42.310 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:42.310 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:42.311 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:42.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-09-21T01:19:42.956 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:43.044 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:43.088 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:43.089 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:43.089 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:43.180 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-09-21T01:19:43.762 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:48 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:43.852 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:43.895 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:43.896 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:43.896 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:44.005 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-09-21T01:19:44.586 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:44.675 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:19.3.0-5120.g18432218.el9.x86_64 is already installed. 2024-09-21T01:19:44.719 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:44.720 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-21T01:19:44.720 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:44.797 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-09-21T01:19:45.375 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:50 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:45.506 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:19.3.0-5120.g18432218.el9 ceph 86 k 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:45.507 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:45.508 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:45.508 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:19:45.508 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:45.508 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 86 k 2024-09-21T01:19:45.508 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 260 k 2024-09-21T01:19:45.508 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:45.712 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-19.3.0-5120.g18432218.el9.x86_64.rpm 423 kB/s | 86 kB 00:00 2024-09-21T01:19:45.713 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:45.713 INFO:teuthology.orchestra.run.smithi090.stdout:Total 420 kB/s | 86 kB 00:00 2024-09-21T01:19:45.713 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:45.720 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:45.720 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:45.783 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:45.783 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:45.929 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:46.458 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:46.731 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:46.731 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:46.731 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:47.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-09-21T01:19:47.645 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:52 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:47.782 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:19.3.0-5120.g18432218.el9 ceph 3.1 M 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:19:47.783 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:47.784 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.1 M 2024-09-21T01:19:47.784 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 15 M 2024-09-21T01:19:47.784 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:48.121 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-19.3.0-5120.g18432218.el9.x86_64.rpm 9.2 MB/s | 3.1 MB 00:00 2024-09-21T01:19:48.122 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:48.122 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-21T01:19:48.123 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:48.131 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:48.131 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:48.213 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:48.213 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:48.731 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:48.757 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:48.757 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T01:19:48.757 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T01:19:48.757 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:19:48.757 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T01:19:48.757 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:50.008 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:50.008 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:50.008 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:50.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-09-21T01:19:50.932 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:55 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:19:51.064 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:19:51.064 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-21T01:19:51.064 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:19.3.0-5120.g18432218.el9 ceph 170 k 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 170 k 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 545 k 2024-09-21T01:19:51.065 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:19:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-19.3.0-5120.g18432218.el9.x86_64.rpm 857 kB/s | 170 kB 00:00 2024-09-21T01:19:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:19:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:Total 850 kB/s | 170 kB 00:00 2024-09-21T01:19:51.265 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:19:51.273 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:19:51.273 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:19:51.338 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:19:51.338 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:19:51.493 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:19:51.603 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:52.407 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 1/1 2024-09-21T01:19:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:52.408 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:19:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:19.3.0-5120.g18432218.el9.x86_64 2024-09-21T01:19:52.408 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:19:52.408 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:19:52.716 DEBUG:teuthology.parallel:result is None 2024-09-21T01:19:52.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:19:52.865 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T01:19:52.891 INFO:teuthology.orchestra.run.smithi045.stdout:19.3.0-5120.g18432218.el9 2024-09-21T01:19:52.892 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T01:19:52.892 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T01:19:52.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:19:53.043 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T01:19:53.068 INFO:teuthology.orchestra.run.smithi090.stdout:19.3.0-5120.g18432218.el9 2024-09-21T01:19:53.069 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5120.g18432218.el9 2024-09-21T01:19:53.069 INFO:teuthology.task.install:The correct ceph version 19.3.0-5120.g18432218 is installed. 2024-09-21T01:19:53.070 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T01:19:53.070 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:19:53.071 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T01:19:53.103 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:19:53.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T01:19:53.138 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T01:19:53.139 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:19:53.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T01:19:53.174 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T01:19:53.243 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:19:53.243 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T01:19:53.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T01:19:53.338 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T01:19:53.338 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:19:53.339 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T01:19:53.368 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T01:19:53.436 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:19:53.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T01:19:53.463 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T01:19:53.530 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T01:19:53.531 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:19:53.531 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T01:19:53.559 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T01:19:53.626 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:19:53.626 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T01:19:53.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T01:19:53.719 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T01:19:53.809 INFO:tasks.cephadm:Config: {'roleless': True, 'cluster-conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 2, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'client mount timeout': 30, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1, 'mon warn on pool no app': False}, 'mon': {'mon op complaint time': 120}, 'osd': {'osd op complaint time': 180}}, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'fs.*is degraded', 'filesystem is degraded', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'filesystem is offline', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'insufficient standby MDS daemons available', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '1843221895adccfb3b692060251dd6a0183bc410'} 2024-09-21T01:19:53.809 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:19:53.809 INFO:tasks.cephadm:Cluster fsid is 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:19:53.810 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T01:19:53.810 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-09-21T01:19:53.810 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi045': '172.21.15.45', 'mon.smithi090': '172.21.15.90'} 2024-09-21T01:19:53.810 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T01:19:53.810 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-09-21T01:19:53.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-09-21T01:19:53.866 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T01:19:53.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:19:54.025 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.4.75+confusa13', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 00:04:17.484514', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '1843221895adccfb3b692060251dd6a0183bc410', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83190/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5120-g18432218', 'node_name': '172.21.2.22+braggi22', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5120.g18432218'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-20 22:54:16.499090', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240920.212106-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T01:19:54.169 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20240920.212106-debug, sha1 1843221895adccfb3b692060251dd6a0183bc410 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:19:54.170 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T01:19:54.170 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm 2024-09-21T01:19:54.170 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T01:19:54.722 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 01:19 /home/ubuntu/cephtest/cephadm 2024-09-21T01:19:54.722 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240920.212106-debug/1843221895adccfb3b692060251dd6a0183bc410/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T01:19:55.245 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 804283 Sep 21 01:19 /home/ubuntu/cephtest/cephadm 2024-09-21T01:19:55.245 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T01:19:55.266 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T01:19:55.304 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 on all hosts... 2024-09-21T01:19:55.305 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T01:19:55.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 pull 2024-09-21T01:19:56.914 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:19:57.703 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout: "repo_digests": [ 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout: ] 2024-09-21T01:20:27.673 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-21T01:20:29.838 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-09-21T01:20:29.838 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:20:29.838 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977", 2024-09-21T01:20:29.838 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-09-21T01:20:29.839 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262" 2024-09-21T01:20:29.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-09-21T01:20:29.839 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-09-21T01:20:29.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-09-21T01:20:29.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-09-21T01:20:29.935 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-09-21T01:20:29.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-09-21T01:20:30.002 INFO:tasks.cephadm:Writing seed config... 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T01:20:30.003 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T01:20:30.004 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:20:30.004 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T01:20:30.033 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9b2fd594-77b7-11ef-baf5-efdc52797490 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T01:20:30.033 DEBUG:teuthology.orchestra.run.smithi045:mon.smithi045> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045.service 2024-09-21T01:20:30.075 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T01:20:30.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 -v bootstrap --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.45 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:20:30.290 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:20:30.291 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', '-v', 'bootstrap', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.45', '--skip-admin-label'] 2024-09-21T01:20:30.291 INFO:teuthology.orchestra.run.smithi045.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T01:20:30.291 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying podman|docker is present... 2024-09-21T01:20:30.315 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 5.2.2 2024-09-21T01:20:30.315 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying lvm2 is present... 2024-09-21T01:20:30.315 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying time synchronization is in place... 2024-09-21T01:20:30.323 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T01:20:30.323 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T01:20:30.330 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T01:20:30.330 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-09-21T01:20:30.337 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-09-21T01:20:30.343 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-09-21T01:20:30.343 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-09-21T01:20:30.343 INFO:teuthology.orchestra.run.smithi045.stdout:Repeating the final host check... 2024-09-21T01:20:30.366 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout 5.2.2 2024-09-21T01:20:30.366 INFO:teuthology.orchestra.run.smithi045.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T01:20:30.366 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl is present 2024-09-21T01:20:30.366 INFO:teuthology.orchestra.run.smithi045.stdout:lvcreate is present 2024-09-21T01:20:30.373 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T01:20:30.374 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T01:20:30.380 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T01:20:30.380 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-09-21T01:20:30.387 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout enabled 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout active 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:Unit chronyd.service is enabled and running 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:Host looks OK 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:Cluster fsid: 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:Acquiring lock 139695838965376 on /run/cephadm/9b2fd594-77b7-11ef-baf5-efdc52797490.lock 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:Lock 139695838965376 acquired on /run/cephadm/9b2fd594-77b7-11ef-baf5-efdc52797490.lock 2024-09-21T01:20:30.394 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 3300 ... 2024-09-21T01:20:30.395 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying IP 172.21.15.45 port 6789 ... 2024-09-21T01:20:30.395 INFO:teuthology.orchestra.run.smithi045.stdout:Base mon IP(s) is [172.21.15.45:3300, 172.21.15.45:6789], mon addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-09-21T01:20:30.398 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.45 metric 100 2024-09-21T01:20:30.398 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2024-09-21T01:20:30.401 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T01:20:30.401 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-21T01:20:30.401 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1769sec hoplimit 64 pref medium 2024-09-21T01:20:30.403 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T01:20:30.403 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T01:20:30.403 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T01:20:30.403 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-21T01:20:30.403 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-21T01:20:30.403 INFO:teuthology.orchestra.run.smithi045.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T01:20:30.404 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-09-21T01:20:30.404 INFO:teuthology.orchestra.run.smithi045.stdout:Mon IP `172.21.15.45` is in CIDR network `172.21.0.0/20` 2024-09-21T01:20:30.404 INFO:teuthology.orchestra.run.smithi045.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T01:20:30.405 INFO:teuthology.orchestra.run.smithi045.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T01:20:30.405 INFO:teuthology.orchestra.run.smithi045.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stdout ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410... 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying blob sha256:8425bc7408d91b180e7fdc62f23e7ae6603757f4f36e9969a4eaf9fd9416b764 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Copying config sha256:ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 2024-09-21T01:20:30.910 INFO:teuthology.orchestra.run.smithi045.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T01:20:31.789 INFO:teuthology.orchestra.run.smithi045.stdout:ceph: stdout ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T01:20:31.790 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph version: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev) 2024-09-21T01:20:31.790 INFO:teuthology.orchestra.run.smithi045.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T01:20:32.771 INFO:teuthology.orchestra.run.smithi045.stdout:stat: stdout 167 167 2024-09-21T01:20:32.771 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial keys... 2024-09-21T01:20:33.609 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQBhH+5mXVhVBRAAaA9fMKSTnJOlTkM5qT652A== 2024-09-21T01:20:34.499 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQBhH+5m+UcKOhAAZWmMO3PY4nNQbKC7eWM2tw== 2024-09-21T01:20:35.354 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph-authtool: stdout AQBiH+5mM5XkMRAAxEGg2edGLQzLILxaj1i+Sg== 2024-09-21T01:20:35.356 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial monmap... 2024-09-21T01:20:36.235 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:monmaptool for smithi045 [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:setting min_mon_release = quincy 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: set fsid to 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:20:36.236 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mon... 2024-09-21T01:20:37.149 INFO:teuthology.orchestra.run.smithi045.stdout:create mon.smithi045 on 2024-09-21T01:20:37.366 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T01:20:37.522 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T01:20:37.715 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490.target. 2024-09-21T01:20:37.716 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490.target. 2024-09-21T01:20:37.989 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045 2024-09-21T01:20:37.989 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045.service: Unit ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045.service not loaded. 2024-09-21T01:20:38.173 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490.target.wants/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045.service → /etc/systemd/system/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@.service. 2024-09-21T01:20:38.228 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 systemd[1]: Starting Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:20:38.505 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 podman[27979]: 2024-09-21 01:20:38.322761145 +0000 UTC m=+0.023000714 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:20:38.505 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 podman[27979]: 2024-09-21 01:20:38.424336458 +0000 UTC m=+0.124576014 container create bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T01:20:38.674 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:20:38.675 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-09-21T01:20:38.682 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:20:38.682 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-09-21T01:20:38.682 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-09-21T01:20:38.682 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T01:20:38.682 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon to start... 2024-09-21T01:20:38.682 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mon... 2024-09-21T01:20:38.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 podman[27979]: 2024-09-21 01:20:38.593196373 +0000 UTC m=+0.293435937 container init bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T01:20:38.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 podman[27979]: 2024-09-21 01:20:38.596462377 +0000 UTC m=+0.296701944 container start bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:20:38.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 ceph-mon[27997]: mkfs 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:38.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 ceph-mon[27997]: mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2024-09-21T01:20:38.889 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 bash[27979]: bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 2024-09-21T01:20:38.889 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:38 smithi045 systemd[1]: Started Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490. 2024-09-21T01:20:39.705 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T01:20:39.705 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout id: 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:39.705 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout services: 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi045 (age 0.514182s) 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout data: 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:mon is available 2024-09-21T01:20:39.706 INFO:teuthology.orchestra.run.smithi045.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: monmap epoch 1 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: last_changed 2024-09-21T01:20:35.691854+0000 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: created 2024-09-21T01:20:35.691854+0000 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: min_mon_release 19 (squid) 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: election_strategy: 1 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: 0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: fsmap 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T01:20:39.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: mgrmap e1: no daemons active 2024-09-21T01:20:39.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:39 smithi045 ceph-mon[27997]: from='client.? 172.21.15.45:0/812115143' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-21T01:20:40.772 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:40 smithi045 ceph-mon[27997]: from='client.? 172.21.15.45:0/2889534435' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T01:20:40.772 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:40 smithi045 ceph-mon[27997]: from='client.? 172.21.15.45:0/2889534435' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T01:20:40.776 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:40.776 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T01:20:40.776 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:40.776 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T01:20:40.777 INFO:teuthology.orchestra.run.smithi045.stdout:Generating new minimal ceph.conf... 2024-09-21T01:20:41.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:41 smithi045 ceph-mon[27997]: from='client.? 172.21.15.45:0/1632872099' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:20:41.824 INFO:teuthology.orchestra.run.smithi045.stdout:Restarting the monitor... 2024-09-21T01:20:42.108 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:41 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:20:42.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:42 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045[27993]: 2024-09-21T01:20:42.106+0000 7fd4ac99d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:20:42.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:42 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045[27993]: 2024-09-21T01:20:42.106+0000 7fd4ac99d640 -1 mon.smithi045@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T01:20:42.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:42 smithi045 podman[28238]: 2024-09-21 01:20:42.250597189 +0000 UTC m=+0.254347554 container died bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T01:20:42.758 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:42 smithi045 podman[28238]: 2024-09-21 01:20:42.573033503 +0000 UTC m=+0.576783866 container cleanup bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:20:42.758 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:42 smithi045 bash[28238]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045 2024-09-21T01:20:43.009 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:42 smithi045 podman[28251]: 2024-09-21 01:20:42.757636745 +0000 UTC m=+0.504374609 container remove bccb7716cb05b8a09f8272d0aaaa58310c6a8727a8a60aecb4a82d9fb769eca3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:20:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 systemd[1]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045.service: Deactivated successfully. 2024-09-21T01:20:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 systemd[1]: Stopped Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490. 2024-09-21T01:20:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 systemd[1]: Starting Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:20:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 podman[28342]: 2024-09-21 01:20:43.123233205 +0000 UTC m=+0.016115929 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:20:43.331 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 podman[28342]: 2024-09-21 01:20:43.23346865 +0000 UTC m=+0.126351399 container create 387dd03e82eb0c7ee167de2f8f55e31436428e4b9d384b3f6f69a5560267fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T01:20:43.506 INFO:teuthology.orchestra.run.smithi045.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 podman[28342]: 2024-09-21 01:20:43.443918146 +0000 UTC m=+0.336800863 container init 387dd03e82eb0c7ee167de2f8f55e31436428e4b9d384b3f6f69a5560267fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 podman[28342]: 2024-09-21 01:20:43.447108659 +0000 UTC m=+0.339991367 container start 387dd03e82eb0c7ee167de2f8f55e31436428e4b9d384b3f6f69a5560267fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: pidfile_write: ignore empty --pid-file 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: load: jerasure load: lrc 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: RocksDB version: 7.9.2 2024-09-21T01:20:43.631 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Git sha 0 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: DB SUMMARY 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: DB Session ID: CV8RALM67XVERDJ82ZC9 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: CURRENT file: CURRENT 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi045/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi045/store.db: 000009.log size: 89538 ; 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.error_if_exists: 0 2024-09-21T01:20:43.632 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.create_if_missing: 0 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.paranoid_checks: 1 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.env: 0x564da3e42d60 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.info_log: 0x564da97bf580 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.statistics: (nil) 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.use_fsync: 0 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_log_file_size: 0 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T01:20:43.633 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.allow_fallocate: 1 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.use_direct_reads: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.db_log_dir: 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.wal_dir: 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T01:20:43.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.write_buffer_manager: 0x564da97c3d60 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.unordered_write: 0 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.row_cache: None 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.wal_filter: None 2024-09-21T01:20:43.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.two_write_queues: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.wal_compression: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.atomic_flush: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.log_readahead_size: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T01:20:43.636 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_background_jobs: 2 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_background_compactions: -1 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_subcompactions: 1 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_open_files: -1 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T01:20:43.637 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_background_flushes: -1 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Compression algorithms supported: 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kZSTD supported: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kXpressCompression supported: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kBZip2Compression supported: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kLZ4Compression supported: 1 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kZlibCompression supported: 1 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: kSnappyCompression supported: 1 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000010 2024-09-21T01:20:43.638 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.merge_operator: 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_filter: None 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564da97bf1a0) 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: cache_index_and_filter_blocks: 1 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: pin_top_level_index_and_filter: 1 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_type: 0 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_block_index_type: 0 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_shortening: 1 2024-09-21T01:20:43.639 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: checksum: 4 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: no_block_cache: 0 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache: 0x564da5783610 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_name: BinnedLRUCache 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_options: 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: capacity : 536870912 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: num_shard_bits : 4 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: strict_capacity_limit : 0 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: high_pri_pool_ratio: 0.000 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_cache_compressed: (nil) 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: persistent_cache: (nil) 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_size: 4096 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_size_deviation: 10 2024-09-21T01:20:43.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_restart_interval: 16 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: index_block_restart_interval: 1 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: metadata_block_size: 4096 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: partition_filters: 0 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: use_delta_encoding: 1 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: filter_policy: bloomfilter 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: whole_key_filtering: 1 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: verify_compression: 0 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: read_amp_bytes_per_bit: 0 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: format_version: 5 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: enable_index_compression: 1 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: block_align: 0 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: max_auto_readahead_size: 262144 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: prepopulate_block_cache: 0 2024-09-21T01:20:43.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: initial_auto_readahead_size: 8192 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression: NoCompression 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T01:20:43.642 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.num_levels: 7 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:20:43.643 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:20:43.644 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T01:20:43.645 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T01:20:43.646 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.inplace_update_support: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.bloom_locality: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.max_successive_merges: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T01:20:43.647 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.ttl: 2592000 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.enable_blob_files: false 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.min_blob_size: 0 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T01:20:43.648 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 107da0c3-9086-4edc-8a0f-e86a496baf41 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726881643483088, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726881643484260, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 86281, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 259, "table_properties": {"data_size": 84428, "index_size": 237, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10119, "raw_average_key_size": 46, "raw_value_size": 78596, "raw_average_value_size": 360, "num_data_blocks": 11, "num_entries": 218, "num_filter_entries": 218, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726881643, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "107da0c3-9086-4edc-8a0f-e86a496baf41", "db_session_id": "CV8RALM67XVERDJ82ZC9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726881643484364, "job": 1, "event": "recovery_finished"} 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi045/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564da97e2a80 2024-09-21T01:20:43.649 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: DB pointer 0x564daa588000 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** DB Stats ** 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2024-09-21T01:20:43.650 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: L0 2/0 86.16 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Sum 2/0 86.16 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 112.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** Compaction Stats [default] ** 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 112.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:20:43.651 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Cumulative compaction: 0.00 GB write, 24.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Interval compaction: 0.00 GB write, 24.01 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Block cache BinnedLRUCache@0x564da5783610#2 capacity: 512.00 MB usage: 1.20 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.78 KB,0.000149012%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T01:20:43.652 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: starting mon.smithi045 rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi045 fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???) e1 preinit fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).mds e1 new map 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).mds e1 print_map 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: e1 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: btime 2024-09-21T01:20:38:644569+0000 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: legacy client fscid: -1 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: No filesystems configured 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:20:43.653 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 bash[28342]: 387dd03e82eb0c7ee167de2f8f55e31436428e4b9d384b3f6f69a5560267fb40 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 systemd[1]: Started Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490. 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: monmap epoch 1 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: last_changed 2024-09-21T01:20:35.691854+0000 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: created 2024-09-21T01:20:35.691854+0000 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: min_mon_release 19 (squid) 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: election_strategy: 1 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: 0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: fsmap 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T01:20:43.654 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:43 smithi045 ceph-mon[28361]: mgrmap e1: no daemons active 2024-09-21T01:20:44.605 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T01:20:44.627 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:20:44.627 INFO:teuthology.orchestra.run.smithi045.stdout:Creating mgr... 2024-09-21T01:20:44.627 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T01:20:44.629 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T01:20:44.629 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying port 0.0.0.0:8443 ... 2024-09-21T01:20:44.887 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mgr.smithi045.txjudz 2024-09-21T01:20:44.887 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mgr.smithi045.txjudz.service: Unit ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mgr.smithi045.txjudz.service not loaded. 2024-09-21T01:20:45.043 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490.target.wants/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mgr.smithi045.txjudz.service → /etc/systemd/system/ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@.service. 2024-09-21T01:20:45.375 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:45 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/747529575' entity='client.admin' 2024-09-21T01:20:45.580 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:20:45.580 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-09-21T01:20:45.587 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:20:45.587 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-09-21T01:20:45.587 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-09-21T01:20:45.587 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T01:20:45.596 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:20:45.596 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-09-21T01:20:45.604 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:20:45.604 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-09-21T01:20:45.604 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-09-21T01:20:45.604 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-09-21T01:20:45.604 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr to start... 2024-09-21T01:20:45.605 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr... 2024-09-21T01:20:46.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:46 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/176150919' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T01:20:46.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:46.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9b2fd594-77b7-11ef-baf5-efdc52797490", 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T01:20:46.729 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T01:20:46.730 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T01:20:38:644569+0000", 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.731 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T01:20:38.645398+0000", 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:46.732 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T01:20:46.733 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:20:46.733 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (1/15)... 2024-09-21T01:20:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:49 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/38643344' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9b2fd594-77b7-11ef-baf5-efdc52797490", 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.851 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:49.852 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.853 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T01:20:38:644569+0000", 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:49.854 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T01:20:38.645398+0000", 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:20:49.855 INFO:teuthology.orchestra.run.smithi045.stdout:mgr not available, waiting (2/15)... 2024-09-21T01:20:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: Activating manager daemon smithi045.txjudz 2024-09-21T01:20:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: mgrmap e2: smithi045.txjudz(active, starting, since 0.00353757s) 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr metadata", "who": "smithi045.txjudz", "id": "smithi045.txjudz"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: Manager daemon smithi045.txjudz is now available 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/trash_purge_schedule"} : dispatch 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' 2024-09-21T01:20:51.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:50 smithi045 ceph-mon[28361]: from='mgr.14100 172.21.15.45:0/2532054822' entity='mgr.smithi045.txjudz' 2024-09-21T01:20:52.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:52 smithi045 ceph-mon[28361]: mgrmap e3: smithi045.txjudz(active, since 1.00796s) 2024-09-21T01:20:52.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:52 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1968227077' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-21T01:20:53.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:20:53.191 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsid": "9b2fd594-77b7-11ef-baf5-efdc52797490", 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 0 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "smithi045" 2024-09-21T01:20:53.192 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T01:20:53.193 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.197 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T01:20:38:644569+0000", 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T01:20:53.198 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ], 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T01:20:38.645398+0000", 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout }, 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:20:53.199 INFO:teuthology.orchestra.run.smithi045.stdout:mgr is available 2024-09-21T01:20:54.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:53 smithi045 ceph-mon[28361]: mgrmap e4: smithi045.txjudz(active, since 2s) 2024-09-21T01:20:56.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:20:56 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2998064623' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout fsid = 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T01:21:02.617 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T01:21:02.618 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling cephadm module... 2024-09-21T01:21:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:05 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1081175921' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-21T01:21:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:06 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1081175921' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T01:21:07.064 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:06 smithi045 ceph-mon[28361]: mgrmap e5: smithi045.txjudz(active, since 15s) 2024-09-21T01:21:08.270 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:08 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2015959681' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T01:21:08.557 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:21:08.557 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T01:21:08.557 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T01:21:08.557 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "smithi045.txjudz", 2024-09-21T01:21:08.557 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T01:21:08.558 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:21:08.558 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-09-21T01:21:08.558 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 5... 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: Active manager daemon smithi045.txjudz restarted 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: Activating manager daemon smithi045.txjudz 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: mgrmap e6: smithi045.txjudz(active, starting, since 0.00370955s) 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr metadata", "who": "smithi045.txjudz", "id": "smithi045.txjudz"} : dispatch 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:21:11.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:11 smithi045 ceph-mon[28361]: Manager daemon smithi045.txjudz is now available 2024-09-21T01:21:12.811 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:21:12.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T01:21:12.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T01:21:12.812 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:21:12.812 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 5 is available 2024-09-21T01:21:12.812 INFO:teuthology.orchestra.run.smithi045.stdout:Verifying orchestrator module is enabled... 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: Found migration_current of "None". Setting to last migration. 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:13.067 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:13.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:21:13.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/trash_purge_schedule"} : dispatch 2024-09-21T01:21:13.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:12 smithi045 ceph-mon[28361]: mgrmap e7: smithi045.txjudz(active, since 1.00595s) 2024-09-21T01:21:14.139 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:13 smithi045 ceph-mon[28361]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T01:21:14.139 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:13 smithi045 ceph-mon[28361]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T01:21:14.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:13 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:12] ENGINE Bus STARTING 2024-09-21T01:21:14.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:13 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:14.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:13 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1709398008' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "orchestrator"} : dispatch 2024-09-21T01:21:14.409 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr module 'orchestrator' is already enabled (always-on) 2024-09-21T01:21:14.409 INFO:teuthology.orchestra.run.smithi045.stdout:Setting orchestrator backend to cephadm... 2024-09-21T01:21:15.071 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:14 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:12] ENGINE Serving on https://172.21.15.45:7150 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:14 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:12] ENGINE Error in HTTPServer.serve 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Traceback (most recent call last): 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._run(expiration_interval) 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:21:15.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: s = self.context.wrap_socket( 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: return self.sslsocket_class._create( 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self.do_handshake() 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._sslobj.do_handshake() 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:14 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:13] ENGINE Serving on http://172.21.15.45:8765 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:14 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:13] ENGINE Bus STARTED 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:14 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1709398008' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "orchestrator"}]': finished 2024-09-21T01:21:15.073 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:14 smithi045 ceph-mon[28361]: mgrmap e8: smithi045.txjudz(active, since 2s) 2024-09-21T01:21:16.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:16 smithi045 ceph-mon[28361]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:16.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:16 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:16.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:16 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:16.728 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T01:21:16.729 INFO:teuthology.orchestra.run.smithi045.stdout:Generating ssh key... 2024-09-21T01:21:17.136 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:17 smithi045 ceph-mon[28361]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:18.743 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:18 smithi045 ceph-mon[28361]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:18.743 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:18 smithi045 ceph-mon[28361]: Generating ssh key... 2024-09-21T01:21:18.743 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:18 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:18.743 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:18 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:19.323 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjGymfmYS8TGbDMZOHlI5j5Fu12nPPgw8EYoUmaocyp5FlNvIm/ebuuwjGEFQv0jckpSMt4/khaHzaM0fxM2sWk1AjLpeUIrfUxwWQvof7qQSI+x+v5EEHpnVlpr+zE7ymg/Z/jat1qpS07Y/wsA4IZFEiMKzKZhsp10nbNYgdEtwrPKPaXrQ5/SmlHhCnL3jECuH1nXp247GInqI/ih5rNaKpKYejX+49nQm423ipgFEA2GjgSTfB35jdYJ55jHyEoJeZ6T2cK37BWaMKTvcJ6B4i5q7eXNLXB/q+aSv6CaT8OAvSMc1GYQy4V5Yi++g1ZJj7FvzRrl4lxpTie1XYSl/p4yJtgX9O96PXWeOpJg7e6u0dDI9kN2K5OcBiqIoibM5+xlwPjF4G6peCrLcyzrkryrQ3s5At89zibnTH0BqDurSKnfAwjR1X6xll6rhPPHtBMP+43kk6/f3bhGJke1c1kYO7WMZ3iIdSDykH74VCdKCFUIYm3PBUB9kq6JU= ceph-9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:21:19.324 INFO:teuthology.orchestra.run.smithi045.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:21:19.324 INFO:teuthology.orchestra.run.smithi045.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T01:21:19.331 INFO:teuthology.orchestra.run.smithi045.stdout:Adding host smithi045... 2024-09-21T01:21:19.656 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:19 smithi045 ceph-mon[28361]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:21.225 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:20 smithi045 ceph-mon[28361]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:22.112 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:21 smithi045 ceph-mon[28361]: Deploying cephadm binary to smithi045 2024-09-21T01:21:24.314 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Added host 'smithi045' with addr '172.21.15.45' 2024-09-21T01:21:24.314 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying mon service with default placement... 2024-09-21T01:21:24.909 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:24 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:24.909 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:24 smithi045 ceph-mon[28361]: Added host smithi045 2024-09-21T01:21:24.909 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:24 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:26.300 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:26 smithi045 ceph-mon[28361]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:26.300 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:26 smithi045 ceph-mon[28361]: Saving service mon spec with placement count:5 2024-09-21T01:21:26.300 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:26 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:26.594 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T01:21:26.594 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying mgr service with default placement... 2024-09-21T01:21:27.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:27 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:27.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:27 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:28.077 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T01:21:28.078 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying crash service with default placement... 2024-09-21T01:21:28.402 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:28 smithi045 ceph-mon[28361]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:28.402 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:28 smithi045 ceph-mon[28361]: Saving service mgr spec with placement count:2 2024-09-21T01:21:28.403 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:28 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:29.293 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-09-21T01:21:29.293 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying ceph-exporter service with default placement... 2024-09-21T01:21:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:29 smithi045 ceph-mon[28361]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:29 smithi045 ceph-mon[28361]: Saving service crash spec with placement * 2024-09-21T01:21:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:29 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:30.774 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-09-21T01:21:30.774 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying prometheus service with default placement... 2024-09-21T01:21:31.032 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:30 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:31.032 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:30 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:31 smithi045 ceph-mon[28361]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:31 smithi045 ceph-mon[28361]: Saving service ceph-exporter spec with placement * 2024-09-21T01:21:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:31 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:31 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:32.694 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-09-21T01:21:32.694 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying grafana service with default placement... 2024-09-21T01:21:32.942 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:32 smithi045 ceph-mon[28361]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:21:32.942 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:32 smithi045 ceph-mon[28361]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:32.942 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:32 smithi045 ceph-mon[28361]: Saving service prometheus spec with placement count:1 2024-09-21T01:21:33.966 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-09-21T01:21:33.967 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying node-exporter service with default placement... 2024-09-21T01:21:34.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:34 smithi045 ceph-mon[28361]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:21:34.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:34 smithi045 ceph-mon[28361]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:34.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:34 smithi045 ceph-mon[28361]: Saving service grafana spec with placement count:1 2024-09-21T01:21:34.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:34 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:35.114 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-09-21T01:21:35.114 INFO:teuthology.orchestra.run.smithi045.stdout:Deploying alertmanager service with default placement... 2024-09-21T01:21:35.762 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:35 smithi045 ceph-mon[28361]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:35.762 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:35 smithi045 ceph-mon[28361]: Saving service node-exporter spec with placement * 2024-09-21T01:21:35.763 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:35 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:36.253 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-09-21T01:21:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:36 smithi045 ceph-mon[28361]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:21:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:36 smithi045 ceph-mon[28361]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:36 smithi045 ceph-mon[28361]: Saving service alertmanager spec with placement count:1 2024-09-21T01:21:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:36 smithi045 ceph-mon[28361]: from='mgr.14118 172.21.15.45:0/1837677291' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:38.072 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:37 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3226880344' entity='client.admin' 2024-09-21T01:21:38.605 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling the dashboard module... 2024-09-21T01:21:39.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:38 smithi045 ceph-mon[28361]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:21:39.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:38 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2510341534' entity='client.admin' 2024-09-21T01:21:40.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:39 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/704976525' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-21T01:21:41.145 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:40 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/704976525' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T01:21:41.145 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:40 smithi045 ceph-mon[28361]: mgrmap e9: smithi045.txjudz(active, since 28s) 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "active_name": "smithi045.txjudz", 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for the mgr to restart... 2024-09-21T01:21:41.761 INFO:teuthology.orchestra.run.smithi045.stdout:Waiting for mgr epoch 9... 2024-09-21T01:21:42.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4072276263' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: Active manager daemon smithi045.txjudz restarted 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: Activating manager daemon smithi045.txjudz 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: mgrmap e10: smithi045.txjudz(active, starting, since 0.00353669s) 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr metadata", "who": "smithi045.txjudz", "id": "smithi045.txjudz"} : dispatch 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:21:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:21:45.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:45 smithi045 ceph-mon[28361]: Manager daemon smithi045.txjudz is now available 2024-09-21T01:21:46.503 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:21:46.503 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:21:46.504 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/trash_purge_schedule"} : dispatch 2024-09-21T01:21:46.504 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:46 smithi045 ceph-mon[28361]: mgrmap e11: smithi045.txjudz(active, since 1.0071s) 2024-09-21T01:21:46.798 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout { 2024-09-21T01:21:46.799 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T01:21:46.799 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T01:21:46.799 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout } 2024-09-21T01:21:46.799 INFO:teuthology.orchestra.run.smithi045.stdout:mgr epoch 9 is available 2024-09-21T01:21:46.799 INFO:teuthology.orchestra.run.smithi045.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-21T01:21:47.418 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:47 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:46] ENGINE Bus STARTING 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:47] ENGINE Serving on http://172.21.15.45:8765 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:47] ENGINE Serving on https://172.21.15.45:7150 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:47] ENGINE Bus STARTED 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: [21/Sep/2024:01:21:47] ENGINE Error in HTTPServer.serve 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Traceback (most recent call last): 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._run(expiration_interval) 2024-09-21T01:21:48.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: s = self.context.wrap_socket( 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: return self.sslsocket_class._create( 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self.do_handshake() 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._sslobj.do_handshake() 2024-09-21T01:21:48.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:21:48.672 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: mgrmap e12: smithi045.txjudz(active, since 2s) 2024-09-21T01:21:48.672 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:48.672 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:48 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:49.460 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE+zCCAuOgAwIBAgIUaLRnwGWwTorEKCreKz+cHUZSWLowDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkyMTAxMjE0OFoXDTM0\nMDkyMjAxMjE0OFowFzEVMBMGA1UEAwwMMTcyLjIxLjE1LjQ1MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAvoUVdY12aHjS66dS2y0i4pVxBXcWAns46blZ\n9zoz/Hgyqp+WmY2nxeUwgi7+k5+MDnBeDWxLsm5bYRWsVpk5FFdhzQvmf6uMpVkR\n4TH28hu0FGeINsCz/oLmeAWd4+Q7d03nt+z3chB41UAVTjqdiam6jyM35HGdGDYh\nCq/SpR6rVkFf6FMiqz+LpsUlSaXyKgSbHVEM4t+p03UqVLYZmIbx280WlzJ42cLX\n4Z9q6vYy2lJmZyLdhBpcPVXUARsW7XGcGfY1oWnkU74/DjlNWqKBU90EiRzxnzK3\n3yghWvSJelL/wy/eaf+ABqAvyRhwDdqmxNZI1XCDUXoJN7ZI3/0OJF4e5dg1HMUH\nqxZaaU2cpCxuZ8S7DQruEVts/Lm5D+4tfUnG/1zcMTlYWwHF6EqYFedUakBUvT1/\n61GJ7ZB1e/V6nSlwViBrE8UkYw1BKxRKHoNRtWgbXharNxWHxhLWNluvfWpgRrXR\nvT1GNsXafb5++Sjxxk+63FVh7s3a+CoiMjtxTYjoyzi+gF/ldshvX0daWgXrhIeu\nLwNhhaDfrX8eLhl8R+5QJM4qAYL2vlnZGuTbTgwH+p7PxZiB0aGS1jXWxafGdcz/\n7Q4GIjOST3ViODYLzqyzXF/p/9CMTBkM6u9AnbONRJh7+IwgKEL7P3alq4NT52HB\n66bUzE8CAwEAAaM/MD0wLQYDVR0RBCYwJIIJc21pdGhpMDQ1ghFkYXNoYm9hcmRf\nc2VydmVyc4cErBUPLTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAP\nMK8bKmuvNMdbJOdiPFvT56Zu8Vee7/iAtdKS4K+Z7DciwdNjDC/K3R/tItSz8T1F\n4DRWwWt9b3UZTb14PQICHkXQqoLI8BhFUEm8kUpT3Jz5SQUP912nk8pfnUFV8H7/\naYnxnfzG4WVYRnSWRSNizxRpMIu86d1shfeJI0h8zrGVvPNnLJ8MtNyXDTcgzq2h\nXDsegvbyxl7ZRZc45oWXq+K60PndB52Hn4MpPqBLxnz7pPPoyYAEWZgDXMlq0rw9\n7z8EykxK/u4W6qwybVq/2+IQs1Dkay0kzdFDYxtoZsM9hZo2Dq7nounJkj+VreJl\nAU8eE9k22FcJNJ65P8oMxrpsjF/nWbJNtcCMINq/UFpI92K9CRmKl/nMVF0BT1sL\nXE+oSh9u9cBnzCdCAAQgRupkgqyLxilXArRqgTyZdhNbxpnvnMfKBZzErzrdc0QV\nbddqetdWSVFycEby71YoSAFF2Rzm1CaQdRxpRBr2aldpgMa8jWqeuF5vqBddIkAK\n/5WraxtO7xvR0gbtoVA/UTQEW+j+6I50nyzx8LJXlR3kggEAUzjjhOFBYPt7RFmE\nnLxI7zNBaZgwVD6hwy7OKkgm+YDhT4E2tRoTJ4Nrf8mbbt2D3n0miDebpCHM5yuT\n6f7Q4CX4/kLZ0Yp6xRy5M/yYbjW2akRjpmx4LThY6w==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKgIBAAKCAgEAvoUVdY12aHjS66dS2y0i4pVxBXcWAns46blZ9zoz/Hgyqp+W\nmY2nxeUwgi7+k5+MDnBeDWxLsm5bYRWsVpk5FFdhzQvmf6uMpVkR4TH28hu0FGeI\nNsCz/oLmeAWd4+Q7d03nt+z3chB41UAVTjqdiam6jyM35HGdGDYhCq/SpR6rVkFf\n6FMiqz+LpsUlSaXyKgSbHVEM4t+p03UqVLYZmIbx280WlzJ42cLX4Z9q6vYy2lJm\nZyLdhBpcPVXUARsW7XGcGfY1oWnkU74/DjlNWqKBU90EiRzxnzK33yghWvSJelL/\nwy/eaf+ABqAvyRhwDdqmxNZI1XCDUXoJN7ZI3/0OJF4e5dg1HMUHqxZaaU2cpCxu\nZ8S7DQruEVts/Lm5D+4tfUnG/1zcMTlYWwHF6EqYFedUakBUvT1/61GJ7ZB1e/V6\nnSlwViBrE8UkYw1BKxRKHoNRtWgbXharNxWHxhLWNluvfWpgRrXRvT1GNsXafb5+\n+Sjxxk+63FVh7s3a+CoiMjtxTYjoyzi+gF/ldshvX0daWgXrhIeuLwNhhaDfrX8e\nLhl8R+5QJM4qAYL2vlnZGuTbTgwH+p7PxZiB0aGS1jXWxafGdcz/7Q4GIjOST3Vi\nODYLzqyzXF/p/9CMTBkM6u9AnbONRJh7+IwgKEL7P3alq4NT52HB66bUzE8CAwEA\nAQKCAgAPNXEvb3naqyfYl1neP0MqqAKxYf6dd21lUwjjMH/tO+x81xKJnYJIIkdL\nYn8Av9rJXZ+pyKcI+RWjBrzkM4UX95sfaD91RPeMTN5IMNnlcoq0mkiIaRJf/d5y\nCa4UqX00b+v7MG0sa4jfKnzI6ur1OK8ONXcvwwC8abhyMZ5KKiGZWCH/hNYeqjWh\nZtmBgJV3/zghZf1LsHkjypWff43u3CQ7MgBcqjMgMb/+teAuHjXpscGU+CE2JiM4\n45IcqHyduypant9MdwyQFtQnyYaVoVAGUX1xiKBNNMTy3obodjRHUeWIP65XSIXV\ngZIZpobDbwdNwqFmeyiF0jfMJ763Z3wDhfpxBOGintDJNePloFDhjsQ3shdk3rDE\nKZ9d2mlzGBkuZ/XrxVY1fb/dni4G8bBUqS57uezzFKO2dyZqZki181q9gUZo2a+J\n8nPgIVv/p5ihCbSXwlVoVTainWNlny+wX1L1xp3CSgCKADqEOwI3N/nfM+zBny+d\nP1mjPo5wbcsroc0w9RjAuLQKdiym81ZvVwuHgKu1xgIEEN41aAb1qUzZuE/BBWjr\nPlodlnrMJ4Ir1dS14ixYdkhJaHQxj6rCXPMQOwApokPHR1+F3rCgEXgSPUFnGojp\nrZsXKnKrhQDBtuv02/rOHaj7qC8V28FEOsSdvNUZZVIu1UxZ2QKCAQEA4tLD4YK+\nxZwjDYCSVVznyqvBGbBA9Vb435VDz08Ug16VFbedFNm2L4Mk9sIZTnnFTDI6tS/V\nUZPyN5sFfMK75+Dvix1/ZETJSf/xO37Dtn8kHSY2GRu/qtSPSclX6vFelJZ2Ud5k\n+JP+wo0lPkWgJdMn9f6vSe7inXA3Cv+JxGRPw/luSh/dG7KXftNl9cqXGv3f9+GZ\nnz1NhmKUPGUzDaKSKXr2Am0vRpTNo/uRH9llaW9kEEzT5r3pR2pZdotOHlfpGK/N\nci3grZqNDsXQMqFWZUeF1b1pPnNILHBG6ZiKJzA0xjQcU/8U3s85FmPGoK73FGHm\nvBBlZfXvTL0FzQKCAQEA1wba90UVxGt+Cq1gjdmpbiLSKONArjGdWzy9L0xOANrn\nVdhrBah7sh6pxtnhTPKI4rzGv+C3bTHotnQmybmhByVHuhtO/u5pINevj/fNDAPT\nJirGv8NrDRgX5E5t6x8B/o1IAW7ZkgCP7P8kQ9/Uc1Cb5P3sQaEP95XNMFdnlpnj\n36V1OjySrojsdTIc5kZFYGusfAoe0LUb5M95jr+ZkKj0HD87bQbog33qCAqXLZRi\n24TfEbp7TzaoxiCGHMNQH6s95lv4pYj1TbcVGy9tN1i9ERN8PZTbsBzvS/QEN0qh\nlWX3LNUwfZFQPini2LDIQPsaoUgIoE4MvIPFDPc+iwKCAQEA3qVYVWILdNnApp5Q\nBa6+b/iMAXU7LhgDDyTrfPD5B8KtwLWNYUwbQyQOApGgWaG6AwNDEVs0n3EDz4VB\nq6dS31eyivGXfIrwssyitG3IGrlOb7Z0X80XJM6lIB6dSCdJWb0fw1jz7/UFm18E\n8c1SWA6fKR6oXYA5vaTKwub3CmoUyh9rHSON3IQ/pORJLy1/iaUuVZmDPAXmtUTK\nkUYdCQNNSZ5DITqfrz6dZJupZoP0wbvvkvh3AAXmmI1R58TyF2LfkbXvimZyJraQ\nQ7H0nd40ehHs01OrnPt4GvbTBLUQ10rAYtYAAItgpxztwoaKEGvuaHl5DZ1iiuUP\n0lmw+QKCAQEAva8xeFH8SsKcdKTPeTLa0OpsxF78ju37pyGkrlW8tepliogYmOww\n+kcsc9t+cZe+g24OUfPVO8F8TX+XO/rB/4C2jolKnX8JREygdfseagvxt0ii/UVR\naUKoyOEa+//ljX1qWMFFsxNF48aLYkjQrE0eyNnrEVYYtMr0MEHagrMMnxzhldcb\nelYVaakkb0rsJou1o9J1rp0llyojTHbe7ArvFP/+10Xx1RU9X4ygIjg1IPtMoOIF\npzJLnkPQqwzs4JL6Tv0xEi4IX18zgkDLMrjYSHkM73IifnB39GO1WGVAt380ldsX\nJJ4M0Fm44HjKnvAkZMQbv/Y6LA59xNtS0wKCAQEAqOINAb88L3StX8zHp9FhXdNC\n3J1Wb2dP2T7H7eiDGqlFS09U3fviiO7DLoxdHm4cUE2e4iz6uwsqXDMuk+6reHsM\nS5C3Icg3oQtmcFm9avZhZb+rS3EvF+5ZcUaxK+oGXxptCK+ap7xAZAnR0u6VdSSe\ngCeawfCTx0wjYuUyeISoPYGXA25zB+np+xjzMD2cPJoG8imMH/tdHjC5r5nHHW7k\nIj3cKRdFWXRbMsh7xzZ9YLtFfZ+M9uL69Z8dZj0k3VaDk4xZ6TESa3SXlyvOw9MQ\ncLGwFVSaP/W3CMSmS0RXR6JXeN2GCOicfCX7+MbYP1EVbJK8DKgng3UTu9tF2A==\n-----END RSA PRIVATE KEY-----\n"} 2024-09-21T01:21:49.709 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:49 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:50.835 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-21T01:21:51.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:51 smithi045 ceph-mon[28361]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:51.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:51 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:52.450 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-21T01:21:52.450 INFO:teuthology.orchestra.run.smithi045.stdout:Creating initial admin user... 2024-09-21T01:21:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:52 smithi045 ceph-mon[28361]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:52 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:53.967 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$xWH.vc6eh1HdsISSLl5xQOMgu9TPi6jYcu74Wqy8wo2cUGB.IXo6C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726881713, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T01:21:53.967 INFO:teuthology.orchestra.run.smithi045.stdout:Fetching dashboard port number... 2024-09-21T01:21:54.573 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:54 smithi045 ceph-mon[28361]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:21:54.573 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:54 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:21:55.115 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T01:21:55.126 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T01:21:55.126 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout disabled 2024-09-21T01:21:55.134 INFO:teuthology.orchestra.run.smithi045.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T01:21:55.134 INFO:teuthology.orchestra.run.smithi045.stdout:systemctl: stdout inactive 2024-09-21T01:21:55.134 INFO:teuthology.orchestra.run.smithi045.stdout:firewalld.service is not enabled 2024-09-21T01:21:55.134 INFO:teuthology.orchestra.run.smithi045.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T01:21:55.135 INFO:teuthology.orchestra.run.smithi045.stdout:Ceph Dashboard is now available at: 2024-09-21T01:21:55.135 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:55.135 INFO:teuthology.orchestra.run.smithi045.stdout: URL: https://smithi045.front.sepia.ceph.com:8443/ 2024-09-21T01:21:55.135 INFO:teuthology.orchestra.run.smithi045.stdout: User: admin 2024-09-21T01:21:55.136 INFO:teuthology.orchestra.run.smithi045.stdout: Password: d5j05odiza 2024-09-21T01:21:55.136 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:55.136 INFO:teuthology.orchestra.run.smithi045.stdout:Saving cluster configuration to /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config directory 2024-09-21T01:21:55.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:55 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3306627019' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-21T01:21:56.528 INFO:teuthology.orchestra.run.smithi045.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T01:21:56.528 INFO:teuthology.orchestra.run.smithi045.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: ceph telemetry on 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout:For more information see: 2024-09-21T01:21:56.529 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.530 INFO:teuthology.orchestra.run.smithi045.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T01:21:56.530 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:21:56.530 INFO:teuthology.orchestra.run.smithi045.stdout:Bootstrap complete. 2024-09-21T01:21:56.570 INFO:tasks.cephadm:Fetching config... 2024-09-21T01:21:56.570 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:21:56.570 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T01:21:56.587 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T01:21:56.587 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:21:56.587 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T01:21:56.643 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T01:21:56.643 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:21:56.643 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/keyring of=/dev/stdout 2024-09-21T01:21:56.709 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T01:21:56.709 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:21:56.709 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T01:21:56.765 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T01:21:56.765 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjGymfmYS8TGbDMZOHlI5j5Fu12nPPgw8EYoUmaocyp5FlNvIm/ebuuwjGEFQv0jckpSMt4/khaHzaM0fxM2sWk1AjLpeUIrfUxwWQvof7qQSI+x+v5EEHpnVlpr+zE7ymg/Z/jat1qpS07Y/wsA4IZFEiMKzKZhsp10nbNYgdEtwrPKPaXrQ5/SmlHhCnL3jECuH1nXp247GInqI/ih5rNaKpKYejX+49nQm423ipgFEA2GjgSTfB35jdYJ55jHyEoJeZ6T2cK37BWaMKTvcJ6B4i5q7eXNLXB/q+aSv6CaT8OAvSMc1GYQy4V5Yi++g1ZJj7FvzRrl4lxpTie1XYSl/p4yJtgX9O96PXWeOpJg7e6u0dDI9kN2K5OcBiqIoibM5+xlwPjF4G6peCrLcyzrkryrQ3s5At89zibnTH0BqDurSKnfAwjR1X6xll6rhPPHtBMP+43kk6/f3bhGJke1c1kYO7WMZ3iIdSDykH74VCdKCFUIYm3PBUB9kq6JU= ceph-9b2fd594-77b7-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T01:21:56.841 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjGymfmYS8TGbDMZOHlI5j5Fu12nPPgw8EYoUmaocyp5FlNvIm/ebuuwjGEFQv0jckpSMt4/khaHzaM0fxM2sWk1AjLpeUIrfUxwWQvof7qQSI+x+v5EEHpnVlpr+zE7ymg/Z/jat1qpS07Y/wsA4IZFEiMKzKZhsp10nbNYgdEtwrPKPaXrQ5/SmlHhCnL3jECuH1nXp247GInqI/ih5rNaKpKYejX+49nQm423ipgFEA2GjgSTfB35jdYJ55jHyEoJeZ6T2cK37BWaMKTvcJ6B4i5q7eXNLXB/q+aSv6CaT8OAvSMc1GYQy4V5Yi++g1ZJj7FvzRrl4lxpTie1XYSl/p4yJtgX9O96PXWeOpJg7e6u0dDI9kN2K5OcBiqIoibM5+xlwPjF4G6peCrLcyzrkryrQ3s5At89zibnTH0BqDurSKnfAwjR1X6xll6rhPPHtBMP+43kk6/f3bhGJke1c1kYO7WMZ3iIdSDykH74VCdKCFUIYm3PBUB9kq6JU= ceph-9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:21:56.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjGymfmYS8TGbDMZOHlI5j5Fu12nPPgw8EYoUmaocyp5FlNvIm/ebuuwjGEFQv0jckpSMt4/khaHzaM0fxM2sWk1AjLpeUIrfUxwWQvof7qQSI+x+v5EEHpnVlpr+zE7ymg/Z/jat1qpS07Y/wsA4IZFEiMKzKZhsp10nbNYgdEtwrPKPaXrQ5/SmlHhCnL3jECuH1nXp247GInqI/ih5rNaKpKYejX+49nQm423ipgFEA2GjgSTfB35jdYJ55jHyEoJeZ6T2cK37BWaMKTvcJ6B4i5q7eXNLXB/q+aSv6CaT8OAvSMc1GYQy4V5Yi++g1ZJj7FvzRrl4lxpTie1XYSl/p4yJtgX9O96PXWeOpJg7e6u0dDI9kN2K5OcBiqIoibM5+xlwPjF4G6peCrLcyzrkryrQ3s5At89zibnTH0BqDurSKnfAwjR1X6xll6rhPPHtBMP+43kk6/f3bhGJke1c1kYO7WMZ3iIdSDykH74VCdKCFUIYm3PBUB9kq6JU= ceph-9b2fd594-77b7-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T01:21:56.900 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjGymfmYS8TGbDMZOHlI5j5Fu12nPPgw8EYoUmaocyp5FlNvIm/ebuuwjGEFQv0jckpSMt4/khaHzaM0fxM2sWk1AjLpeUIrfUxwWQvof7qQSI+x+v5EEHpnVlpr+zE7ymg/Z/jat1qpS07Y/wsA4IZFEiMKzKZhsp10nbNYgdEtwrPKPaXrQ5/SmlHhCnL3jECuH1nXp247GInqI/ih5rNaKpKYejX+49nQm423ipgFEA2GjgSTfB35jdYJ55jHyEoJeZ6T2cK37BWaMKTvcJ6B4i5q7eXNLXB/q+aSv6CaT8OAvSMc1GYQy4V5Yi++g1ZJj7FvzRrl4lxpTie1XYSl/p4yJtgX9O96PXWeOpJg7e6u0dDI9kN2K5OcBiqIoibM5+xlwPjF4G6peCrLcyzrkryrQ3s5At89zibnTH0BqDurSKnfAwjR1X6xll6rhPPHtBMP+43kk6/f3bhGJke1c1kYO7WMZ3iIdSDykH74VCdKCFUIYm3PBUB9kq6JU= ceph-9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:21:56.912 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T01:21:57.163 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:21:57.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:56 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2057736348' entity='client.admin' 2024-09-21T01:21:59.050 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:58 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3835914664' entity='client.admin' 2024-09-21T01:21:59.051 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:21:58 smithi045 ceph-mon[28361]: mgrmap e13: smithi045.txjudz(active, since 12s) 2024-09-21T01:21:59.080 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T01:21:59.080 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T01:21:59.338 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:22:01.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:01.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:01.238 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"} : dispatch 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: Updating smithi045:/etc/ceph/ceph.conf 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: Updating smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: Updating smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:01.239 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi045", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]} : dispatch 2024-09-21T01:22:01.240 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi045", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-21T01:22:01.240 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:01 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:01.268 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi090 2024-09-21T01:22:01.268 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:22:01.268 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.conf 2024-09-21T01:22:01.288 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:22:01.288 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:22:01.346 INFO:tasks.cephadm:Adding host smithi090 to orchestrator... 2024-09-21T01:22:01.346 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch host add smithi090 2024-09-21T01:22:02.041 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:22:02.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:02 smithi045 ceph-mon[28361]: Deploying daemon ceph-exporter.smithi045 on smithi045 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]} : dispatch 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:04.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:04 smithi045 ceph-mon[28361]: Deploying daemon crash.smithi045 on smithi045 2024-09-21T01:22:05.847 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:05 smithi045 ceph-mon[28361]: Deploying cephadm binary to smithi090 2024-09-21T01:22:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:06 smithi045 ceph-mon[28361]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:06 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:06 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:06 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:06 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:06 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:07.052 INFO:teuthology.orchestra.run.smithi045.stdout:Added host 'smithi090' with addr '172.21.15.90' 2024-09-21T01:22:07.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:07 smithi045 ceph-mon[28361]: Deploying daemon node-exporter.smithi045 on smithi045 2024-09-21T01:22:07.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:07 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:07.719 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T01:22:07.977 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:22:08.872 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:22:08.872 INFO:teuthology.orchestra.run.smithi045.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}] 2024-09-21T01:22:08.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:08 smithi045 ceph-mon[28361]: Added host smithi090 2024-09-21T01:22:08.873 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:08 smithi045 ceph-mon[28361]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:09 smithi045 ceph-mon[28361]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:22:10.126 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T01:22:10.126 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-21T01:22:10.564 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:22:10.717 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:10 smithi045 ceph-mon[28361]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:10.717 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:10 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:10.717 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:10 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:10.717 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:10 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:10.717 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:10 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:11.631 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-09-21T01:22:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:11 smithi045 ceph-mon[28361]: Deploying daemon alertmanager.smithi045 on smithi045 2024-09-21T01:22:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:11 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:11 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1410888980' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-21T01:22:12.815 INFO:tasks.cephadm:Adding mon.smithi045 on smithi045 2024-09-21T01:22:12.815 INFO:tasks.cephadm:Adding mon.smithi090 on smithi090 2024-09-21T01:22:12.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch apply mon '2;smithi045:172.21.15.45=smithi045;smithi090:172.21.15.90=smithi090' 2024-09-21T01:22:12.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:12 smithi045 ceph-mon[28361]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:12.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:12 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1410888980' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T01:22:12.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:12 smithi045 ceph-mon[28361]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T01:22:13.060 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:13.107 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:13.672 INFO:teuthology.orchestra.run.smithi090.stdout:Scheduled mon update... 2024-09-21T01:22:14.168 DEBUG:teuthology.orchestra.run.smithi090:mon.smithi090> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi090.service 2024-09-21T01:22:14.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:14.171 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:14.440 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:14.485 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:15.134 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:15.134 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:15.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:14 smithi045 ceph-mon[28361]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:15.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:14 smithi045 ceph-mon[28361]: from='client.14199 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi045:172.21.15.45=smithi045;smithi090:172.21.15.90=smithi090", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:22:15.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:14 smithi045 ceph-mon[28361]: Saving service mon spec with placement smithi045:172.21.15.45=smithi045;smithi090:172.21.15.90=smithi090;count:2 2024-09-21T01:22:15.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:14 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:16.015 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:15 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/2977373435' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:16.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:16.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:16.949 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:16 smithi045 ceph-mon[28361]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:16.954 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:16.998 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:17.655 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:17.655 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:18.006 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:17 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3709494855' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:18.978 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:18 smithi045 ceph-mon[28361]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:19.190 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:19.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:19.434 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:19.479 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:20.102 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:20.103 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:20.103 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:20.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:20 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/4019756334' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:21.653 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:21.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:21.895 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"} : dispatch 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:21 smithi045 ceph-mon[28361]: Deploying daemon grafana.smithi045 on smithi045 2024-09-21T01:22:21.939 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:22.622 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:22.622 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:22.622 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:22 smithi045 ceph-mon[28361]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:22 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/257308787' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:24.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:24.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:24.428 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:24.473 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:25.110 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:25.110 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:25.110 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:25.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:24 smithi045 ceph-mon[28361]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:26.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:25 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/2587092014' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:26.695 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:26.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:26.960 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:27.004 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:27.070 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:26 smithi045 ceph-mon[28361]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:27.646 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:27.646 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:27.646 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:27 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3771193802' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:28.789 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:28 smithi045 ceph-mon[28361]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:29.182 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:29.182 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:29.422 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:29.467 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:30.125 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:30.125 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:30.125 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:31.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:30 smithi045 ceph-mon[28361]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:31.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:30 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3649326423' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:31.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:31.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:31.917 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:31.961 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:32.596 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:32.596 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:32.596 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:32.705 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:32.705 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.705 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.705 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.705 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.705 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:32.706 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:32 smithi045 ceph-mon[28361]: Deploying daemon prometheus.smithi045 on smithi045 2024-09-21T01:22:33.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:33 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/2374518391' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:34.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:34.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:34.400 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:34.445 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:34.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:34 smithi045 ceph-mon[28361]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:35.072 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:35.072 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:35.072 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:35.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:35 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/2557630943' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:36.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:36.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:36.868 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:36.912 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:36.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:36 smithi045 ceph-mon[28361]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:36.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:36 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:37.587 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:37.587 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:37.587 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:37.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:37 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/655460490' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:38.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:38 smithi045 ceph-mon[28361]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:39.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:39.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:39.387 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:39.431 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:40.062 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:40.063 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:40.063 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:40.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:40 smithi045 ceph-mon[28361]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:40.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:40 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/4241404597' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:41.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:41.625 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:41.868 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:41.912 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:42.551 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:42.551 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:42.551 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:42.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:42 smithi045 ceph-mon[28361]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:42.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:42 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/560469407' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:44.109 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:44.109 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:44.350 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:44.394 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:44.888 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:44 smithi045 ceph-mon[28361]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:45.047 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:45.047 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:45.048 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:45.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:45 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/121237423' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:46.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:46.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:46.867 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:46.911 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:46.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:46 smithi045 ceph-mon[28361]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:22:46.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:46.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:46.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:46.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:46 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr module enable", "module": "prometheus"} : dispatch 2024-09-21T01:22:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:47.560 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:47.560 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:47 smithi045 ceph-mon[28361]: from='mgr.14164 172.21.15.45:0/2097037200' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T01:22:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:47 smithi045 ceph-mon[28361]: mgrmap e14: smithi045.txjudz(active, since 61s) 2024-09-21T01:22:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:47 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3336725892' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:49.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:49.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:49.365 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:49.410 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:50.083 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:50.083 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:50.083 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:50.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:50 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/597403896' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:51.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:51.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:51.930 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:51.976 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: Active manager daemon smithi045.txjudz restarted 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: Activating manager daemon smithi045.txjudz 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: mgrmap e15: smithi045.txjudz(active, starting, since 0.00436354s) 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr metadata", "who": "smithi045.txjudz", "id": "smithi045.txjudz"} : dispatch 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T01:22:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:51 smithi045 ceph-mon[28361]: Manager daemon smithi045.txjudz is now available 2024-09-21T01:22:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:52.670 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:52.671 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/mirror_snapshot_schedule"} : dispatch 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.txjudz/trash_purge_schedule"} : dispatch 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3435079035' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:53.279 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:52 smithi045 ceph-mon[28361]: mgrmap e16: smithi045.txjudz(active, since 1.0077s) 2024-09-21T01:22:54.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:54.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:54.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:54 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:22:54.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:54.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:54.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:54.473 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:54.523 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T01:22:55.794 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:55.794 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:55.794 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:55 smithi045 ceph-mon[28361]: [21/Sep/2024:01:22:54] ENGINE Bus STARTING 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:55 smithi045 ceph-mon[28361]: [21/Sep/2024:01:22:54] ENGINE Serving on http://172.21.15.45:8765 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:55 smithi045 ceph-mon[28361]: mgrmap e17: smithi045.txjudz(active, since 2s) 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:55 smithi045 ceph-mon[28361]: [21/Sep/2024:01:22:54] ENGINE Serving on https://172.21.15.45:7150 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:55 smithi045 ceph-mon[28361]: [21/Sep/2024:01:22:54] ENGINE Bus STARTED 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:55 smithi045 ceph-mon[28361]: [21/Sep/2024:01:22:54] ENGINE Error in HTTPServer.serve 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: Traceback (most recent call last): 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T01:22:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._connections.run(self.expiration_interval) 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._run(expiration_interval) 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: s = self.context.wrap_socket( 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: return self.sslsocket_class._create( 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self.do_handshake() 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T01:22:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: self._sslobj.do_handshake() 2024-09-21T01:22:55.922 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T01:22:56.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:56 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/2744000839' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:22:56.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd/host:smithi045", "name": "osd_memory_target"} : dispatch 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"} : dispatch 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:22:57.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:22:57.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:22:58.179 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:22:58.847 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: Updating smithi045:/etc/ceph/ceph.conf 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: Updating smithi090:/etc/ceph/ceph.conf 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: Updating smithi090:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: Updating smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]} : dispatch 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-21T01:22:58.848 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:22:58.884 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:22:58.884 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:22:58.885 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:22:59.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:59 smithi045 ceph-mon[28361]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:22:59.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:59 smithi045 ceph-mon[28361]: Updating smithi045:/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:22:59.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:59 smithi045 ceph-mon[28361]: Updating smithi090:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:22:59.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:59 smithi045 ceph-mon[28361]: Updating smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T01:22:59.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:59 smithi045 ceph-mon[28361]: Deploying daemon ceph-exporter.smithi090 on smithi090 2024-09-21T01:22:59.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:22:59 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/4061504130' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:01.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:23:01.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:23:01.412 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:02.122 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:23:02.122 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:23:02.122 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:23:02.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:02.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:02.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:02.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:02.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]} : dispatch 2024-09-21T01:23:02.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-21T01:23:02.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:02.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: Deploying daemon crash.smithi090 on smithi090 2024-09-21T01:23:02.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:02 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/257912490' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:03.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:03 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:04.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:23:04.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:23:04.958 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:05.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:05.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:05.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:05.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:05.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:05 smithi045 ceph-mon[28361]: Deploying daemon node-exporter.smithi090 on smithi090 2024-09-21T01:23:05.638 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:23:05.638 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:23:05.639 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:23:06.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:06 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3720541739' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:07.473 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:23:07.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:23:07.713 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:09.046 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:23:09.046 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:23:09.046 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mgr.smithi090.zsfvyu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.zsfvyu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: Deploying daemon mgr.smithi090.zsfvyu on smithi090 2024-09-21T01:23:09.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:09 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/2261302572' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:10.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:23:10.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:23:11.664 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:12.434 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:23:12.434 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":1,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:20:35.691854Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T01:23:12.434 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 1 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: Deploying daemon mon.smithi090 on smithi090 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:13.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:12 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/1131037520' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:13.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:13 smithi045 ceph-mon[28361]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:14.698 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-21T01:23:14.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-21T01:23:15.152 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:23:15.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 systemd[1]: Starting Ceph mon.smithi090 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:23:15.616 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 podman[35098]: 2024-09-21 01:23:15.49162529 +0000 UTC m=+0.095601942 container create a7fe49195229eab87bec6279d5b7d101ac2a669ffe3fa9e3dde08fe8e3dd9fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731) 2024-09-21T01:23:15.616 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 podman[35098]: 2024-09-21 01:23:15.41308754 +0000 UTC m=+0.017064195 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:23:16.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 podman[35098]: 2024-09-21 01:23:15.895967462 +0000 UTC m=+0.499944129 container init a7fe49195229eab87bec6279d5b7d101ac2a669ffe3fa9e3dde08fe8e3dd9fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:23:16.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 podman[35098]: 2024-09-21 01:23:15.899631872 +0000 UTC m=+0.503608535 container start a7fe49195229eab87bec6279d5b7d101ac2a669ffe3fa9e3dde08fe8e3dd9fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T01:23:16.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T01:23:16.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev), process ceph-mon, pid 2 2024-09-21T01:23:16.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: pidfile_write: ignore empty --pid-file 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: load: jerasure load: lrc 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: RocksDB version: 7.9.2 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Git sha 0 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Compile date 2024-09-20 22:29:49 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: DB SUMMARY 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: DB Session ID: QT41H3BBEIJHEJQPYHB7 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: CURRENT file: CURRENT 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi090/store.db dir, Total Num: 0, files: 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi090/store.db: 000004.log size: 511 ; 2024-09-21T01:23:16.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.error_if_exists: 0 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.create_if_missing: 0 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.paranoid_checks: 1 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.env: 0x564e5b34dd60 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T01:23:16.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.info_log: 0x564e605f1580 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.statistics: (nil) 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.use_fsync: 0 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_log_file_size: 0 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.allow_fallocate: 1 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T01:23:16.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.use_direct_reads: 0 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.db_log_dir: 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.wal_dir: 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.write_buffer_manager: 0x564e605f5d60 2024-09-21T01:23:16.166 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.unordered_write: 0 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.row_cache: None 2024-09-21T01:23:16.167 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.wal_filter: None 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.two_write_queues: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.wal_compression: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.atomic_flush: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.log_readahead_size: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T01:23:16.168 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_background_jobs: 2 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_background_compactions: -1 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_subcompactions: 1 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T01:23:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_open_files: -1 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_background_flushes: -1 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Compression algorithms supported: 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kZSTD supported: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kXpressCompression supported: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kBZip2Compression supported: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kLZ4Compression supported: 1 2024-09-21T01:23:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kZlibCompression supported: 1 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: kSnappyCompression supported: 1 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000005 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.merge_operator: 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_filter: None 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T01:23:16.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564e605f11a0) 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: index_type: 0 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: data_block_index_type: 0 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: index_shortening: 1 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: checksum: 4 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: no_block_cache: 0 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache: 0x564e5c5b5610 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache_options: 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: capacity : 536870912 2024-09-21T01:23:16.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: num_shard_bits : 4 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: strict_capacity_limit : 0 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache_compressed: (nil) 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: persistent_cache: (nil) 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_size: 4096 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_size_deviation: 10 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_restart_interval: 16 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: index_block_restart_interval: 1 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: metadata_block_size: 4096 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: partition_filters: 0 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: use_delta_encoding: 1 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: filter_policy: bloomfilter 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: whole_key_filtering: 1 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: verify_compression: 0 2024-09-21T01:23:16.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: format_version: 5 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: enable_index_compression: 1 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_align: 0 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: max_auto_readahead_size: 262144 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: prepopulate_block_cache: 0 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: initial_auto_readahead_size: 8192 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression: NoCompression 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T01:23:16.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.num_levels: 7 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:23:16.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T01:23:16.176 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T01:23:16.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.inplace_update_support: 0 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T01:23:16.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.bloom_locality: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.max_successive_merges: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.ttl: 2592000 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.enable_blob_files: false 2024-09-21T01:23:16.179 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.min_blob_size: 0 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7101ef04-2eb5-4178-bc5f-d853e202607a 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726881795935747, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T01:23:16.180 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726881795936180, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726881795, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7101ef04-2eb5-4178-bc5f-d853e202607a", "db_session_id": "QT41H3BBEIJHEJQPYHB7", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726881795936274, "job": 1, "event": "recovery_finished"} 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi090/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564e60614a80 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: DB pointer 0x564e613ba000 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** DB Stats ** 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:23:16.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** Compaction Stats [default] ** 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** Compaction Stats [default] ** 2024-09-21T01:23:16.182 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T01:23:16.183 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Block cache BinnedLRUCache@0x564e5c5b5610#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090 does not exist in monmap, will attempt to join an existing cluster 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: using public_addr v2:172.21.15.90:0/0 -> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: starting mon.smithi090 rank -1 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi090 fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(???) e0 preinit fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).mds e1 new map 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).mds e1 print_map 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: e1 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: btime 2024-09-21T01:20:38:644569+0000 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T01:23:16.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: legacy client fscid: -1 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: No filesystems configured 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T01:23:16.185 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: Deploying daemon node-exporter.smithi090 on smithi090 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/3720541739' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mgr.smithi090.zsfvyu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.zsfvyu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T01:23:16.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: Deploying daemon mgr.smithi090.zsfvyu on smithi090 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/2261302572' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: Deploying daemon mon.smithi090 on smithi090 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/1131037520' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:16.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 ceph-mon[35142]: mon.smithi090@-1(synchronizing).paxosservice(auth 1..8) refresh upgraded, format 0 -> 3 2024-09-21T01:23:16.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 bash[35098]: a7fe49195229eab87bec6279d5b7d101ac2a669ffe3fa9e3dde08fe8e3dd9fd0 2024-09-21T01:23:16.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:15 smithi090 systemd[1]: Started Ceph mon.smithi090 for 9b2fd594-77b7-11ef-baf5-efdc52797490. 2024-09-21T01:23:16.188 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:15 smithi045 ceph-mon[28361]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:20.983 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:23:20.983 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":2,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","modified":"2024-09-21T01:23:15.967546Z","created":"2024-09-21T01:20:35.691854Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T01:23:20.983 INFO:teuthology.orchestra.run.smithi090.stderr:dumped monmap epoch 2 2024-09-21T01:23:21.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:21.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:23:21.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: mon.smithi045 calling monitor election 2024-09-21T01:23:21.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/497764709' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/smithi090.zsfvyu/crt"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: mon.smithi090 calling monitor election 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: mon.smithi045 is new leader, mons smithi045,smithi090 in quorum (ranks 0,1) 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: monmap epoch 2 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: last_changed 2024-09-21T01:23:15.967546+0000 2024-09-21T01:23:21.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: created 2024-09-21T01:20:35.691854+0000 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: min_mon_release 19 (squid) 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: election_strategy: 1 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: 0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.smithi090 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: fsmap 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: mgrmap e17: smithi045.txjudz(active, since 29s) 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: overall HEALTH_OK 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: Standby manager daemon smithi090.zsfvyu started 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/smithi090.zsfvyu/key"} : dispatch 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:23:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:21.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:23:21.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: mon.smithi045 calling monitor election 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/497764709' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/smithi090.zsfvyu/crt"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: mon.smithi090 calling monitor election 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: mon.smithi045 is new leader, mons smithi045,smithi090 in quorum (ranks 0,1) 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: monmap epoch 2 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: last_changed 2024-09-21T01:23:15.967546+0000 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: created 2024-09-21T01:20:35.691854+0000 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: min_mon_release 19 (squid) 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: election_strategy: 1 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: 0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-09-21T01:23:21.422 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: 1: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.smithi090 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: fsmap 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: osdmap e5: 0 total, 0 up, 0 in 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: mgrmap e17: smithi045.txjudz(active, since 29s) 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: overall HEALTH_OK 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: Standby manager daemon smithi090.zsfvyu started 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/smithi090.zsfvyu/key"} : dispatch 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.? 172.21.15.90:0/2101220613' entity='mgr.smithi090.zsfvyu' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-21T01:23:21.423 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.424 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.424 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:21.605 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T01:23:21.605 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T01:23:21.877 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:22 smithi090 ceph-mon[35142]: mgrmap e18: smithi045.txjudz(active, since 29s), standbys: smithi090.zsfvyu 2024-09-21T01:23:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr metadata", "who": "smithi090.zsfvyu", "id": "smithi090.zsfvyu"} : dispatch 2024-09-21T01:23:22.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:23:22.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:22.249 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:22 smithi045 ceph-mon[28361]: mgrmap e18: smithi045.txjudz(active, since 29s), standbys: smithi090.zsfvyu 2024-09-21T01:23:22.249 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr metadata", "who": "smithi090.zsfvyu", "id": "smithi090.zsfvyu"} : dispatch 2024-09-21T01:23:22.249 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:23:22.249 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:23:22.545 INFO:teuthology.orchestra.run.smithi045.stdout:# minimal ceph.conf for 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:23:22.545 INFO:teuthology.orchestra.run.smithi045.stdout:[global] 2024-09-21T01:23:22.545 INFO:teuthology.orchestra.run.smithi045.stdout: fsid = 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:23:22.545 INFO:teuthology.orchestra.run.smithi045.stdout: mon_host = [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] 2024-09-21T01:23:23.201 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T01:23:23.201 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:23:23.201 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T01:23:23.231 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:23:23.231 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:23:23.292 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:23 smithi045 ceph-mon[28361]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:23.293 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:23.293 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:23.293 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:23 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3151415334' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:23.297 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:23:23.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T01:23:23.319 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:23 smithi090 ceph-mon[35142]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:23.319 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:23.319 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:23.319 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:23 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3151415334' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:23.323 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:23:23.323 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T01:23:23.387 INFO:tasks.cephadm:setting config [client] client_mount_timeout = 600 2024-09-21T01:23:23.387 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set client client_mount_timeout 600 2024-09-21T01:23:23.389 INFO:tasks.cephadm:setting config [client] debug_client = 20 2024-09-21T01:23:23.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set client debug_client 20 2024-09-21T01:23:23.432 INFO:tasks.cephadm:setting config [client] debug_ms = 1 2024-09-21T01:23:23.433 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set client debug_ms 1 2024-09-21T01:23:23.476 INFO:tasks.cephadm:setting config [client] rados_mon_op_timeout = 900 2024-09-21T01:23:23.476 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set client rados_mon_op_timeout 900 2024-09-21T01:23:23.519 INFO:tasks.cephadm:setting config [client] rados_osd_op_timeout = 900 2024-09-21T01:23:23.519 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set client rados_osd_op_timeout 900 2024-09-21T01:23:23.563 INFO:tasks.cephadm:setting config [mds] debug_mds = 20 2024-09-21T01:23:23.563 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds 20 2024-09-21T01:23:23.606 INFO:tasks.cephadm:setting config [mds] debug_mds_balancer = 20 2024-09-21T01:23:23.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds debug_mds_balancer 20 2024-09-21T01:23:23.650 INFO:tasks.cephadm:setting config [mds] debug_ms = 2 2024-09-21T01:23:23.650 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds debug_ms 2 2024-09-21T01:23:23.696 INFO:tasks.cephadm:setting config [mds] mds_debug_frag = True 2024-09-21T01:23:23.696 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_frag True 2024-09-21T01:23:23.739 INFO:tasks.cephadm:setting config [mds] mds_debug_scatterstat = True 2024-09-21T01:23:23.739 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds mds_debug_scatterstat True 2024-09-21T01:23:23.792 INFO:tasks.cephadm:setting config [mds] mds_op_complaint_time = 180 2024-09-21T01:23:23.792 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds mds_op_complaint_time 180 2024-09-21T01:23:23.808 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:23.818 INFO:tasks.cephadm:setting config [mds] mds_verify_scatter = True 2024-09-21T01:23:23.818 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds mds_verify_scatter True 2024-09-21T01:23:23.865 INFO:tasks.cephadm:setting config [mds] osd_op_complaint_time = 180 2024-09-21T01:23:23.865 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds osd_op_complaint_time 180 2024-09-21T01:23:23.918 INFO:tasks.cephadm:setting config [mds] rados_mon_op_timeout = 900 2024-09-21T01:23:23.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds rados_mon_op_timeout 900 2024-09-21T01:23:23.965 INFO:tasks.cephadm:setting config [mds] rados_osd_op_timeout = 900 2024-09-21T01:23:23.965 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mds rados_osd_op_timeout 900 2024-09-21T01:23:23.968 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:23.972 INFO:tasks.cephadm:setting config [mgr] client_mount_timeout = 30 2024-09-21T01:23:23.972 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mgr client_mount_timeout 30 2024-09-21T01:23:24.021 INFO:tasks.cephadm:setting config [mgr] debug_client = 20 2024-09-21T01:23:24.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mgr debug_client 20 2024-09-21T01:23:24.075 INFO:tasks.cephadm:setting config [mgr] debug_mgr = 20 2024-09-21T01:23:24.075 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mgr debug_mgr 20 2024-09-21T01:23:24.128 INFO:tasks.cephadm:setting config [mgr] debug_ms = 1 2024-09-21T01:23:24.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mgr debug_ms 1 2024-09-21T01:23:24.187 INFO:tasks.cephadm:setting config [mgr] mon_warn_on_pool_no_app = False 2024-09-21T01:23:24.187 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mgr mon_warn_on_pool_no_app False 2024-09-21T01:23:24.237 INFO:tasks.cephadm:setting config [mon] mon_op_complaint_time = 120 2024-09-21T01:23:24.237 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set mon mon_op_complaint_time 120 2024-09-21T01:23:24.294 INFO:tasks.cephadm:setting config [osd] osd_op_complaint_time = 180 2024-09-21T01:23:24.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config set osd osd_op_complaint_time 180 2024-09-21T01:23:24.352 DEBUG:tasks.cephadm:set 22 configs 2024-09-21T01:23:24.352 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'client_mount_timeout', '600'], check_status=True, hostname='smithi045') 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: Updating smithi045:/etc/ceph/ceph.conf 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: Updating smithi090:/etc/ceph/ceph.conf 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: Updating smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: Updating smithi090:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.353 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: Reconfiguring mon.smithi045 (unknown last config time)... 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:24.354 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:24 smithi045 ceph-mon[28361]: Reconfiguring daemon mon.smithi045 on smithi045 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: Updating smithi045:/etc/ceph/ceph.conf 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: Updating smithi090:/etc/ceph/ceph.conf 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: Updating smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:24.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: Updating smithi090:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: Reconfiguring mon.smithi045 (unknown last config time)... 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:24.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:24 smithi090 ceph-mon[35142]: Reconfiguring daemon mon.smithi045 on smithi045 2024-09-21T01:23:25.356 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:25.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:25 smithi045 ceph-mon[28361]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:25.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:25 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2089793049' entity='client.admin' 2024-09-21T01:23:25.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:25 smithi090 ceph-mon[35142]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:25.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:25 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2089793049' entity='client.admin' 2024-09-21T01:23:26.267 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_client', '20'], check_status=True, hostname='smithi045') 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 -- 172.21.15.45:0/1180984879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f0ef807aec0 msgr2=0x7f0ef807d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 --2- 172.21.15.45:0/1180984879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f0ef807aec0 0x7f0ef807d3d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000c00 tx=0x7f0ef40058d0 comp rx=0 tx=0).stop 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 -- 172.21.15.45:0/1180984879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f0c102340 msgr2=0x7f0f0c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 --2- 172.21.15.45:0/1180984879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f0c102340 0x7f0f0c197ff0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00d6f0 tx=0x7f0efc00dbb0 comp rx=0 tx=0).stop 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 -- 172.21.15.45:0/1180984879 shutdown_connections 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 --2- 172.21.15.45:0/1180984879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f0ef807aec0 0x7f0ef807d3d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 --2- 172.21.15.45:0/1180984879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0f0c103470 0x7f0f0c198660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 --2- 172.21.15.45:0/1180984879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0f0c102340 0x7f0f0c197ff0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 -- 172.21.15.45:0/1180984879 >> 172.21.15.45:0/1180984879 conn(0x7f0f0c0fdcd0 msgr2=0x7f0f0c106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 -- 172.21.15.45:0/1180984879 shutdown_connections 2024-09-21T01:23:27.422 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:27.421+0000 7f0f10a32640 1 -- 172.21.15.45:0/1180984879 wait complete. 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: Reconfiguring mgr.smithi045.txjudz (unknown last config time)... 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mgr.smithi045.txjudz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: Reconfiguring daemon mgr.smithi045.txjudz on smithi045 2024-09-21T01:23:27.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:27 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1180984879' entity='client.admin' 2024-09-21T01:23:28.121 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: Reconfiguring mgr.smithi045.txjudz (unknown last config time)... 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mgr.smithi045.txjudz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: Reconfiguring daemon mgr.smithi045.txjudz on smithi045 2024-09-21T01:23:28.122 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:27 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1180984879' entity='client.admin' 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.035+0000 7f669c314640 1 -- 172.21.15.45:0/675982446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669406b5b0 msgr2=0x7f6694105c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.035+0000 7f669c314640 1 --2- 172.21.15.45:0/675982446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669406b5b0 0x7f6694105c80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6690009880 tx=0x7f669002a5d0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 -- 172.21.15.45:0/675982446 shutdown_connections 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 --2- 172.21.15.45:0/675982446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f669406b5b0 0x7f6694105c80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 -- 172.21.15.45:0/675982446 >> 172.21.15.45:0/675982446 conn(0x7f66940fd8a0 msgr2=0x7f66940ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 -- 172.21.15.45:0/675982446 shutdown_connections 2024-09-21T01:23:29.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 -- 172.21.15.45:0/675982446 wait complete. 2024-09-21T01:23:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 Processor -- start 2024-09-21T01:23:29.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.036+0000 7f669c314640 1 -- start start 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f669c314640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f669406b5b0 0x7f6694198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f669c314640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 0x7f6694199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f669c314640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6694108c50 con 0x7f6694198dd0 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f669c314640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6694108ad0 con 0x7f669406b5b0 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 0x7f6694199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 0x7f6694199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36766/0 (socket says 172.21.15.45:36766) 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 -- 172.21.15.45:0/803985071 learned_addr learned my addr 172.21.15.45:0/803985071 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f669a0b0640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f669406b5b0 0x7f6694198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 -- 172.21.15.45:0/803985071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f669406b5b0 msgr2=0x7f6694198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f669406b5b0 0x7f6694198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 -- 172.21.15.45:0/803985071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6694199ea0 con 0x7f6694198dd0 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.037+0000 7f66998af640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 0x7f6694199960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f668400b4a0 tx=0x7f668400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:29.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.038+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6684014070 con 0x7f6694198dd0 2024-09-21T01:23:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.038+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f669419a190 con 0x7f6694198dd0 2024-09-21T01:23:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.038+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(22 keys) ==== 976+0+0 (secure 0 0 0) 0x7f6684010a40 con 0x7f6694198dd0 2024-09-21T01:23:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.038+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6684017960 con 0x7f6694198dd0 2024-09-21T01:23:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.038+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f669419a7a0 con 0x7f6694198dd0 2024-09-21T01:23:29.039 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.039+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66941093d0 con 0x7f6694198dd0 2024-09-21T01:23:29.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.040+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f6684017070 con 0x7f6694198dd0 2024-09-21T01:23:29.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.040+0000 7f668b7fe640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f668007b010 0x7f668007d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.040+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f668400d040 con 0x7f6694198dd0 2024-09-21T01:23:29.040 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.040+0000 7f669a0b0640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f668007b010 0x7f668007d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.041 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.041+0000 7f669a0b0640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f668007b010 0x7f668007d520 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f66900096c0 tx=0x7f6690005a20 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:29.045 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.045+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f668401e030 con 0x7f6694198dd0 2024-09-21T01:23:29.083 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.088 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.089 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.095 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.105 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.105 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.115 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.118 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.126 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.131 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.131 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.144 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:29.336 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.336+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f669419e9f0 con 0x7f6694198dd0 2024-09-21T01:23:29.340 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.340+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v13)=0 v13) ==== 114+0+0 (secure 0 0 0) 0x7f6684068c00 con 0x7f6694198dd0 2024-09-21T01:23:29.342 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.341+0000 7f668b7fe640 1 -- 172.21.15.45:0/803985071 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f6684063c30 con 0x7f6694198dd0 2024-09-21T01:23:29.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f668007b010 msgr2=0x7f668007d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f668007b010 0x7f668007d520 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f66900096c0 tx=0x7f6690005a20 comp rx=0 tx=0).stop 2024-09-21T01:23:29.343 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 msgr2=0x7f6694199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 0x7f6694199960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f668400b4a0 tx=0x7f668400b960 comp rx=0 tx=0).stop 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 shutdown_connections 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f668007b010 0x7f668007d520 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6694198dd0 0x7f6694199960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 --2- 172.21.15.45:0/803985071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f669406b5b0 0x7f6694198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 >> 172.21.15.45:0/803985071 conn(0x7f66940fd8a0 msgr2=0x7f66940fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 shutdown_connections 2024-09-21T01:23:29.344 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.343+0000 7f669c314640 1 -- 172.21.15.45:0/803985071 wait complete. 2024-09-21T01:23:29.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.857+0000 7f6c46712640 1 -- 172.21.15.45:0/2449026121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 msgr2=0x7f6c40108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.857+0000 7f6c46712640 1 --2- 172.21.15.45:0/2449026121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40108b00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009880 tx=0x7f6c3402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.857+0000 7f6c46712640 1 -- 172.21.15.45:0/2449026121 shutdown_connections 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.857+0000 7f6c46712640 1 --2- 172.21.15.45:0/2449026121 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40108b00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.857+0000 7f6c46712640 1 --2- 172.21.15.45:0/2449026121 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c4006c110 0x7f6c40106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.858+0000 7f6c46712640 1 -- 172.21.15.45:0/2449026121 >> 172.21.15.45:0/2449026121 conn(0x7f6c400fdcd0 msgr2=0x7f6c40100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.858+0000 7f6c46712640 1 -- 172.21.15.45:0/2449026121 shutdown_connections 2024-09-21T01:23:29.859 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.858+0000 7f6c46712640 1 -- 172.21.15.45:0/2449026121 wait complete. 2024-09-21T01:23:29.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.858+0000 7f6c46712640 1 Processor -- start 2024-09-21T01:23:29.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.858+0000 7f6c46712640 1 -- start start 2024-09-21T01:23:29.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c46712640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c4006c110 0x7f6c40103650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c46712640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40101f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c46712640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4010bf60 con 0x7f6c40106620 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c46712640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4010bde0 con 0x7f6c4006c110 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40101f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3f7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40101f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36776/0 (socket says 172.21.15.45:36776) 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3f7fe640 1 -- 172.21.15.45:0/4102979879 learned_addr learned my addr 172.21.15.45:0/4102979879 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3ffff640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c4006c110 0x7f6c40103650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3f7fe640 1 -- 172.21.15.45:0/4102979879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c4006c110 msgr2=0x7f6c40103650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3f7fe640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c4006c110 0x7f6c40103650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.859+0000 7f6c3f7fe640 1 -- 172.21.15.45:0/4102979879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c40103e60 con 0x7f6c40106620 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.860+0000 7f6c3f7fe640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40101f90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005940 tx=0x7f6c34004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.860+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6c3403f070 con 0x7f6c40106620 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.860+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c4019cf90 con 0x7f6c40106620 2024-09-21T01:23:29.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.860+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f6c34003a40 con 0x7f6c40106620 2024-09-21T01:23:29.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.860+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6c34035030 con 0x7f6c40106620 2024-09-21T01:23:29.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.860+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c401026a0 con 0x7f6c40106620 2024-09-21T01:23:29.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.861+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c4010c6b0 con 0x7f6c40106620 2024-09-21T01:23:29.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.862+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f6c34042070 con 0x7f6c40106620 2024-09-21T01:23:29.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.863+0000 7f6c3d7fa640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6c2807aec0 0x7f6c2807d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:29.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.863+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f6c340bdfd0 con 0x7f6c40106620 2024-09-21T01:23:29.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.863+0000 7f6c3ffff640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6c2807aec0 0x7f6c2807d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:29.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.864+0000 7f6c3ffff640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6c2807aec0 0x7f6c2807d3d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000abe0 tx=0x7f6c3000a2d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:29.868 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:29.868+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6c3408a8b0 con 0x7f6c40106620 2024-09-21T01:23:29.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:29 smithi090 ceph-mon[35142]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:29.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:29.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:29.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi045", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]} : dispatch 2024-09-21T01:23:29.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:29.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:29 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/803985071' entity='client.admin' 2024-09-21T01:23:30.059 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.058+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f6c40104030 con 0x7f6c40106620 2024-09-21T01:23:30.059 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:29 smithi045 ceph-mon[28361]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:30.059 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:30.059 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:30.059 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi045", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]} : dispatch 2024-09-21T01:23:30.059 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:30.060 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:29 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/803985071' entity='client.admin' 2024-09-21T01:23:30.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.063+0000 7f6c3d7fa640 1 -- 172.21.15.45:0/4102979879 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v14)=0 v14) ==== 108+0+0 (secure 0 0 0) 0x7f6c3408f760 con 0x7f6c40106620 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.066+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6c2807aec0 msgr2=0x7f6c2807d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.066+0000 7f6c46712640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6c2807aec0 0x7f6c2807d3d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000abe0 tx=0x7f6c3000a2d0 comp rx=0 tx=0).stop 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.066+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 msgr2=0x7f6c40101f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.066+0000 7f6c46712640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40101f90 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c34005940 tx=0x7f6c34004fa0 comp rx=0 tx=0).stop 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.066+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 shutdown_connections 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.067+0000 7f6c46712640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6c2807aec0 0x7f6c2807d3d0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6c3000abe0 tx=0x7f6c3000a2d0 comp rx=0 tx=0).stop 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.067+0000 7f6c46712640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6c40106620 0x7f6c40101f90 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:30.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.067+0000 7f6c46712640 1 --2- 172.21.15.45:0/4102979879 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6c4006c110 0x7f6c40103650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:30.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.067+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 >> 172.21.15.45:0/4102979879 conn(0x7f6c400fdcd0 msgr2=0x7f6c40100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:30.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.067+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 shutdown_connections 2024-09-21T01:23:30.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:30.067+0000 7f6c46712640 1 -- 172.21.15.45:0/4102979879 wait complete. 2024-09-21T01:23:30.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:30 smithi090 ceph-mon[35142]: Reconfiguring ceph-exporter.smithi045 (monmap changed)... 2024-09-21T01:23:30.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:30 smithi090 ceph-mon[35142]: Reconfiguring daemon ceph-exporter.smithi045 on smithi045 2024-09-21T01:23:30.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:30 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4102979879' entity='client.admin' 2024-09-21T01:23:31.055 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:30 smithi045 ceph-mon[28361]: Reconfiguring ceph-exporter.smithi045 (monmap changed)... 2024-09-21T01:23:31.055 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:30 smithi045 ceph-mon[28361]: Reconfiguring daemon ceph-exporter.smithi045 on smithi045 2024-09-21T01:23:31.055 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:30 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4102979879' entity='client.admin' 2024-09-21T01:23:31.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 -- 172.21.15.45:0/1263296967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 msgr2=0x7fb3fc108ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 --2- 172.21.15.45:0/1263296967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc108ae0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009880 tx=0x7fb3f002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 -- 172.21.15.45:0/1263296967 shutdown_connections 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 --2- 172.21.15.45:0/1263296967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc108ae0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 --2- 172.21.15.45:0/1263296967 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3fc06b6b0 0x7fb3fc106140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 -- 172.21.15.45:0/1263296967 >> 172.21.15.45:0/1263296967 conn(0x7fb3fc0fdce0 msgr2=0x7fb3fc100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 -- 172.21.15.45:0/1263296967 shutdown_connections 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.257+0000 7fb40220a640 1 -- 172.21.15.45:0/1263296967 wait complete. 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb40220a640 1 Processor -- start 2024-09-21T01:23:31.291 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb40220a640 1 -- start start 2024-09-21T01:23:31.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb40220a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3fc06b6b0 0x7fb3fc103700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb40220a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc102040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb40220a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc10bf90 con 0x7fb3fc106680 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb40220a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb3fc10be10 con 0x7fb3fc06b6b0 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb3faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb3faffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36834/0 (socket says 172.21.15.45:36834) 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb3faffd640 1 -- 172.21.15.45:0/3100887857 learned_addr learned my addr 172.21.15.45:0/3100887857 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.258+0000 7fb3fb7fe640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3fc06b6b0 0x7fb3fc103700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb3faffd640 1 -- 172.21.15.45:0/3100887857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3fc06b6b0 msgr2=0x7fb3fc103700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb3faffd640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3fc06b6b0 0x7fb3fc103700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb3faffd640 1 -- 172.21.15.45:0/3100887857 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3fc103f10 con 0x7fb3fc106680 2024-09-21T01:23:31.294 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb3faffd640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc102040 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0005ab0 tx=0x7fb3f0004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb3f003f070 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7fb3f0003a40 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.259+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3fc102640 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.260+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb3f0035030 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.260+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3fc102b20 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.261+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fb3f0042070 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.261+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3fc10c6e0 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.262+0000 7fb3f8ff9640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb3e407af10 0x7fb3e407d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.263+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb3f00bd7d0 con 0x7fb3fc106680 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.263+0000 7fb3fb7fe640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb3e407af10 0x7fb3e407d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.264+0000 7fb3fb7fe640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb3e407af10 0x7fb3e407d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec005fd0 tx=0x7fb3ec00b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:31.295 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.267+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb3f0003bf0 con 0x7fb3fc106680 2024-09-21T01:23:31.483 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.482+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fb3fc102dd0 con 0x7fb3fc106680 2024-09-21T01:23:31.487 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.486+0000 7fb3f8ff9640 1 -- 172.21.15.45:0/3100887857 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v17)=0 v17) ==== 120+0+0 (secure 0 0 0) 0x7fb3f007fe20 con 0x7fb3fc106680 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb3e407af10 msgr2=0x7fb3e407d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb3e407af10 0x7fb3e407d420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec005fd0 tx=0x7fb3ec00b390 comp rx=0 tx=0).stop 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 msgr2=0x7fb3fc102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc102040 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0005ab0 tx=0x7fb3f0004fa0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 shutdown_connections 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb3e407af10 0x7fb3e407d420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.489 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb3fc106680 0x7fb3fc102040 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 --2- 172.21.15.45:0/3100887857 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3fc06b6b0 0x7fb3fc103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 >> 172.21.15.45:0/3100887857 conn(0x7fb3fc0fdce0 msgr2=0x7fb3fc100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:31.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 shutdown_connections 2024-09-21T01:23:31.490 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.488+0000 7fb40220a640 1 -- 172.21.15.45:0/3100887857 wait complete. 2024-09-21T01:23:31.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 -- 172.21.15.45:0/3178160274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628106900 msgr2=0x7ff628108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 --2- 172.21.15.45:0/3178160274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628106900 0x7ff628108d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009880 tx=0x7ff61c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 -- 172.21.15.45:0/3178160274 shutdown_connections 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 --2- 172.21.15.45:0/3178160274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628106900 0x7ff628108d20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 --2- 172.21.15.45:0/3178160274 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff628101820 0x7ff628101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 -- 172.21.15.45:0/3178160274 >> 172.21.15.45:0/3178160274 conn(0x7ff6280fd450 msgr2=0x7ff6280ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 -- 172.21.15.45:0/3178160274 shutdown_connections 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.072+0000 7ff62e77c640 1 -- 172.21.15.45:0/3178160274 wait complete. 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.073+0000 7ff62e77c640 1 Processor -- start 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.073+0000 7ff62e77c640 1 -- start start 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff62e77c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 0x7ff628199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff62e77c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff628106900 0x7ff6281999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff62e77c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff62810bff0 con 0x7ff628101820 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff62e77c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff62810be70 con 0x7ff628106900 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff627fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 0x7ff628199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff627fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 0x7ff628199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36796/0 (socket says 172.21.15.45:36796) 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff627fff640 1 -- 172.21.15.45:0/1050431435 learned_addr learned my addr 172.21.15.45:0/1050431435 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:31.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff6277fe640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff628106900 0x7ff6281999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff627fff640 1 -- 172.21.15.45:0/1050431435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff628106900 msgr2=0x7ff6281999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff627fff640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff628106900 0x7ff6281999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.074+0000 7ff627fff640 1 -- 172.21.15.45:0/1050431435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62819a4c0 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.075+0000 7ff627fff640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 0x7ff628199380 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff61800b420 tx=0x7ff61800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.075+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff618011630 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.075+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7ff618016070 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.075+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62819a7b0 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.075+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff618010a60 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.075+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62819ad90 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.076+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62810c740 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.077+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7ff618010c20 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.077+0000 7ff6257fa640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff61007af10 0x7ff61007d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.077+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff61809c370 con 0x7ff628101820 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.077+0000 7ff6277fe640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff61007af10 0x7ff61007d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:31.770 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.078+0000 7ff6277fe640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff61007af10 0x7ff61007d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000c00 tx=0x7ff61c0095e0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.082+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff618068c50 con 0x7ff628101820 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.276+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mds_op_complaint_time}] v 0) -- 0x7ff62819b0a0 con 0x7ff628101820 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.279+0000 7ff6257fa640 1 -- 172.21.15.45:0/1050431435 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_op_complaint_time}]=0 v15)=0 v15) ==== 121+0+0 (secure 0 0 0) 0x7ff61806db00 con 0x7ff628101820 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff61007af10 msgr2=0x7ff61007d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff61007af10 0x7ff61007d420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff61c000c00 tx=0x7ff61c0095e0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 msgr2=0x7ff628199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 0x7ff628199380 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff61800b420 tx=0x7ff61800b8e0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 shutdown_connections 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff61007af10 0x7ff61007d420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff628106900 0x7ff6281999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 --2- 172.21.15.45:0/1050431435 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff628101820 0x7ff628199380 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 >> 172.21.15.45:0/1050431435 conn(0x7ff6280fd450 msgr2=0x7ff6280ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 shutdown_connections 2024-09-21T01:23:31.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.283+0000 7ff62e77c640 1 -- 172.21.15.45:0/1050431435 wait complete. 2024-09-21T01:23:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:31 smithi090 ceph-mon[35142]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:31 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1050431435' entity='client.admin' 2024-09-21T01:23:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:31 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2797269132' entity='client.admin' 2024-09-21T01:23:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:31 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3100887857' entity='client.admin' 2024-09-21T01:23:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:31 smithi045 ceph-mon[28361]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:31 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1050431435' entity='client.admin' 2024-09-21T01:23:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:31 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2797269132' entity='client.admin' 2024-09-21T01:23:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:31 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3100887857' entity='client.admin' 2024-09-21T01:23:32.527 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'debug_ms', '1'], check_status=True, hostname='smithi045') 2024-09-21T01:23:32.527 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi045') 2024-09-21T01:23:32.765 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 -- 172.21.15.45:0/1939826799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 msgr2=0x7f51581027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 --2- 172.21.15.45:0/1939826799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f51581027b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f5154009880 tx=0x7f515402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 -- 172.21.15.45:0/1939826799 shutdown_connections 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 --2- 172.21.15.45:0/1939826799 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5158103470 0x7f5158103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 --2- 172.21.15.45:0/1939826799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f51581027b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 -- 172.21.15.45:0/1939826799 >> 172.21.15.45:0/1939826799 conn(0x7f51580fdcb0 msgr2=0x7f5158100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 -- 172.21.15.45:0/1939826799 shutdown_connections 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.150+0000 7f515f612640 1 -- 172.21.15.45:0/1939826799 wait complete. 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515f612640 1 Processor -- start 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515f612640 1 -- start start 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515f612640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f515806bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.766 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515f612640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5158103470 0x7f51581a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515f612640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f515810ae80 con 0x7f5158102340 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515f612640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f515810ad00 con 0x7f5158103470 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515d3ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f515806bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515d3ae640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f515806bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36814/0 (socket says 172.21.15.45:36814) 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515d3ae640 1 -- 172.21.15.45:0/2797269132 learned_addr learned my addr 172.21.15.45:0/2797269132 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515cbad640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5158103470 0x7f51581a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515d3ae640 1 -- 172.21.15.45:0/2797269132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5158103470 msgr2=0x7f51581a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515d3ae640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5158103470 0x7f51581a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.151+0000 7f515d3ae640 1 -- 172.21.15.45:0/2797269132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f515806c490 con 0x7f5158102340 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.152+0000 7f515d3ae640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f515806bdd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5154004c90 tx=0x7f5154004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.152+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f515403f070 con 0x7f5158102340 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.152+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7f5154005170 con 0x7f5158102340 2024-09-21T01:23:32.767 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.152+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51581a0960 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.152+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f5154035030 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.152+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51581a0e10 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.153+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515810b5d0 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.154+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f5154042070 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.154+0000 7f514e7fc640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f514407aec0 0x7f514407d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.154+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f51540bd7c0 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.155+0000 7f515cbad640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f514407aec0 0x7f514407d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.155+0000 7f515cbad640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f514407aec0 0x7f514407d3d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5148009bd0 tx=0x7f5148005f90 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.159+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f515408a0c0 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.358+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_mds_balancer}] v 0) -- 0x7f515802e840 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.363+0000 7f514e7fc640 1 -- 172.21.15.45:0/2797269132 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds_balancer}]=0 v16)=0 v16) ==== 117+0+0 (secure 0 0 0) 0x7f515408ef70 con 0x7f5158102340 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f514407aec0 msgr2=0x7f514407d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f514407aec0 0x7f514407d3d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5148009bd0 tx=0x7f5148005f90 comp rx=0 tx=0).stop 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 msgr2=0x7f515806bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f515806bdd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5154004c90 tx=0x7f5154004d70 comp rx=0 tx=0).stop 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 shutdown_connections 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f514407aec0 0x7f514407d3d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5158103470 0x7f51581a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 --2- 172.21.15.45:0/2797269132 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5158102340 0x7f515806bdd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 >> 172.21.15.45:0/2797269132 conn(0x7f51580fdcb0 msgr2=0x7f51581068d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 shutdown_connections 2024-09-21T01:23:32.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:31.366+0000 7f515f612640 1 -- 172.21.15.45:0/2797269132 wait complete. 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.166+0000 7fbbfab8c640 1 -- 172.21.15.45:0/756369990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf4102a10 msgr2=0x7fbbf4102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.166+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/756369990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf4102a10 0x7fbbf4102e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009880 tx=0x7fbbe402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 -- 172.21.15.45:0/756369990 shutdown_connections 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/756369990 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf41057e0 0x7fbbf4107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/756369990 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf4102a10 0x7fbbf4102e60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 -- 172.21.15.45:0/756369990 >> 172.21.15.45:0/756369990 conn(0x7fbbf40fd490 msgr2=0x7fbbf40ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:33.169 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 -- 172.21.15.45:0/756369990 shutdown_connections 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 -- 172.21.15.45:0/756369990 wait complete. 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 Processor -- start 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.167+0000 7fbbfab8c640 1 -- start start 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbfab8c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf4102a10 0x7fbbf419b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbfab8c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 0x7fbbf419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbfab8c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf410ae50 con 0x7fbbf41057e0 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbfab8c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf410acd0 con 0x7fbbf4102a10 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 0x7fbbf419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 0x7fbbf419bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36850/0 (socket says 172.21.15.45:36850) 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 -- 172.21.15.45:0/985635148 learned_addr learned my addr 172.21.15.45:0/985635148 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:33.170 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf8928640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf4102a10 0x7fbbf419b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:33.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 -- 172.21.15.45:0/985635148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf4102a10 msgr2=0x7fbbf419b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf4102a10 0x7fbbf419b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 -- 172.21.15.45:0/985635148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf419c5d0 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf3fff640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 0x7fbbf419bb00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00c8a0 tx=0x7fbbdc00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbbdc0079b0 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.168+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(23 keys) ==== 1001+0+0 (secure 0 0 0) 0x7fbbdc00f450 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.169+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf419c8c0 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.169+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbbdc018630 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.169+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf419ce20 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.170+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf410b5a0 con 0x7fbbf41057e0 2024-09-21T01:23:33.172 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.171+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fbbdc007b50 con 0x7fbbf41057e0 2024-09-21T01:23:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.171+0000 7fbbf1ffb640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbbe007af10 0x7fbbe007d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.171+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fbbdc09cdd0 con 0x7fbbf41057e0 2024-09-21T01:23:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.171+0000 7fbbf8928640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbbe007af10 0x7fbbe007d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:33.173 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.172+0000 7fbbf8928640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbbe007af10 0x7fbbe007d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009230 tx=0x7fbbe4000bc0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:33.178 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.177+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbbdc014030 con 0x7fbbf41057e0 2024-09-21T01:23:33.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.368+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_osd_op_timeout}] v 0) -- 0x7fbbf419d1a0 con 0x7fbbf41057e0 2024-09-21T01:23:33.373 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.372+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_osd_op_timeout}]=0 v18)=0 v18) ==== 123+0+0 (secure 0 0 0) 0x7fbbdc00f600 con 0x7fbbf41057e0 2024-09-21T01:23:33.375 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.373+0000 7fbbf1ffb640 1 -- 172.21.15.45:0/985635148 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fbbdc007e40 con 0x7fbbf41057e0 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbbe007af10 msgr2=0x7fbbe007d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbbe007af10 0x7fbbe007d420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009230 tx=0x7fbbe4000bc0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 msgr2=0x7fbbf419bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 0x7fbbf419bb00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00c8a0 tx=0x7fbbdc00cd60 comp rx=0 tx=0).stop 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 shutdown_connections 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbbe007af10 0x7fbbe007d420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbbf41057e0 0x7fbbf419bb00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 --2- 172.21.15.45:0/985635148 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbbf4102a10 0x7fbbf419b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 >> 172.21.15.45:0/985635148 conn(0x7fbbf40fd490 msgr2=0x7fbbf40fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 shutdown_connections 2024-09-21T01:23:33.377 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.375+0000 7fbbfab8c640 1 -- 172.21.15.45:0/985635148 wait complete. 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.683+0000 7fb6a1093640 1 -- 172.21.15.45:0/3507026004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 msgr2=0x7fb69c108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.683+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3507026004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c108b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb684009880 tx=0x7fb68402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.683+0000 7fb6a1093640 1 -- 172.21.15.45:0/3507026004 shutdown_connections 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.683+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3507026004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c108b20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.683+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3507026004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb69c06c290 0x7fb69c1060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.683+0000 7fb6a1093640 1 -- 172.21.15.45:0/3507026004 >> 172.21.15.45:0/3507026004 conn(0x7fb69c0fdcd0 msgr2=0x7fb69c100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:33.685 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.684+0000 7fb6a1093640 1 -- 172.21.15.45:0/3507026004 shutdown_connections 2024-09-21T01:23:33.686 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.684+0000 7fb6a1093640 1 -- 172.21.15.45:0/3507026004 wait complete. 2024-09-21T01:23:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.684+0000 7fb6a1093640 1 Processor -- start 2024-09-21T01:23:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb6a1093640 1 -- start start 2024-09-21T01:23:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb6a1093640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb69c06c290 0x7fb69c199310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb6a1093640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb6a1093640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c10c0c0 con 0x7fb69c106640 2024-09-21T01:23:33.687 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb6a1093640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c10bf40 con 0x7fb69c06c290 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69259c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36858/0 (socket says 172.21.15.45:36858) 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69259c640 1 -- 172.21.15.45:0/3599952680 learned_addr learned my addr 172.21.15.45:0/3599952680 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69ad9d640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb69c06c290 0x7fb69c199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69259c640 1 -- 172.21.15.45:0/3599952680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb69c06c290 msgr2=0x7fb69c199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69259c640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb69c06c290 0x7fb69c199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.685+0000 7fb69259c640 1 -- 172.21.15.45:0/3599952680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c19a450 con 0x7fb69c106640 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.686+0000 7fb69259c640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c199980 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb684005ab0 tx=0x7fb684004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.686+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb68403f070 con 0x7fb69c106640 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.686+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69c19a6e0 con 0x7fb69c106640 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.686+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(24 keys) ==== 1032+0+0 (secure 0 0 0) 0x7fb684003a60 con 0x7fb69c106640 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.686+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb684035030 con 0x7fb69c106640 2024-09-21T01:23:33.688 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.686+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69c19ac60 con 0x7fb69c106640 2024-09-21T01:23:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.687+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb69c10c810 con 0x7fb69c106640 2024-09-21T01:23:33.689 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.688+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fb684042070 con 0x7fb69c106640 2024-09-21T01:23:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.688+0000 7fb698d99640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb68807aec0 0x7fb68807d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.689+0000 7fb69ad9d640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb68807aec0 0x7fb68807d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.689+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fb6840bda00 con 0x7fb69c106640 2024-09-21T01:23:33.690 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.689+0000 7fb69ad9d640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb68807aec0 0x7fb68807d3d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00abe0 tx=0x7fb68c00a2d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:33.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.696+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb684003c10 con 0x7fb69c106640 2024-09-21T01:23:33.889 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.888+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fb69c1040a0 con 0x7fb69c106640 2024-09-21T01:23:33.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.892+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v19)=0 v19) ==== 123+0+0 (secure 0 0 0) 0x7fb68408a300 con 0x7fb69c106640 2024-09-21T01:23:33.894 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.893+0000 7fb698d99640 1 -- 172.21.15.45:0/3599952680 <== mon.0 v2:172.21.15.45:3300/0 8 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb684085330 con 0x7fb69c106640 2024-09-21T01:23:33.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb68807aec0 msgr2=0x7fb68807d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb68807aec0 0x7fb68807d3d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00abe0 tx=0x7fb68c00a2d0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 msgr2=0x7fb69c199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:33.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c199980 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb684005ab0 tx=0x7fb684004270 comp rx=0 tx=0).stop 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 shutdown_connections 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb68807aec0 0x7fb68807d3d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb69c106640 0x7fb69c199980 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 --2- 172.21.15.45:0/3599952680 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb69c06c290 0x7fb69c199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 >> 172.21.15.45:0/3599952680 conn(0x7fb69c0fdcd0 msgr2=0x7fb69c100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 shutdown_connections 2024-09-21T01:23:33.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:33.895+0000 7fb6a1093640 1 -- 172.21.15.45:0/3599952680 wait complete. 2024-09-21T01:23:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:33 smithi090 ceph-mon[35142]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:33.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:33 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/985635148' entity='client.admin' 2024-09-21T01:23:34.159 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:33 smithi045 ceph-mon[28361]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:34.160 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:33 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/985635148' entity='client.admin' 2024-09-21T01:23:35.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:34 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3599952680' entity='client.admin' 2024-09-21T01:23:35.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:34 smithi045 ceph-mon[28361]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:35.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:34 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3599952680' entity='client.admin' 2024-09-21T01:23:35.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:34 smithi090 ceph-mon[35142]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 -- 172.21.15.45:0/393823729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4102350 msgr2=0x7fa1b41027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/393823729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4102350 0x7fa1b41027c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009880 tx=0x7fa1ac02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 -- 172.21.15.45:0/393823729 shutdown_connections 2024-09-21T01:23:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/393823729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b4103480 0x7fa1b4103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/393823729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4102350 0x7fa1b41027c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 -- 172.21.15.45:0/393823729 >> 172.21.15.45:0/393823729 conn(0x7fa1b40fdce0 msgr2=0x7fa1b4100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:35.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 -- 172.21.15.45:0/393823729 shutdown_connections 2024-09-21T01:23:35.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 -- 172.21.15.45:0/393823729 wait complete. 2024-09-21T01:23:35.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.270+0000 7fa1bc18b640 1 Processor -- start 2024-09-21T01:23:35.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1bc18b640 1 -- start start 2024-09-21T01:23:35.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1bc18b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b4102350 0x7fa1b4199580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:35.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1bc18b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 0x7fa1b4199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:35.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1bc18b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410b170 con 0x7fa1b4103480 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1bc18b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b410aff0 con 0x7fa1b4102350 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 0x7fa1b4199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 0x7fa1b4199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36870/0 (socket says 172.21.15.45:36870) 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 -- 172.21.15.45:0/1849129118 learned_addr learned my addr 172.21.15.45:0/1849129118 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9f27640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b4102350 0x7fa1b4199580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 -- 172.21.15.45:0/1849129118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b4102350 msgr2=0x7fa1b4199580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b4102350 0x7fa1b4199580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 -- 172.21.15.45:0/1849129118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b419a6c0 con 0x7fa1b4103480 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.271+0000 7fa1b9726640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 0x7fa1b4199bf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800b420 tx=0x7fa1a800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.272+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa1a8011630 con 0x7fa1b4103480 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.272+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa1a8016070 con 0x7fa1b4103480 2024-09-21T01:23:35.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.272+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b419a9b0 con 0x7fa1b4103480 2024-09-21T01:23:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.272+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa1a8010b80 con 0x7fa1b4103480 2024-09-21T01:23:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.272+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b419af90 con 0x7fa1b4103480 2024-09-21T01:23:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.273+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b4102840 con 0x7fa1b4103480 2024-09-21T01:23:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.274+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fa1a8010d40 con 0x7fa1b4103480 2024-09-21T01:23:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.274+0000 7fa1b2ffd640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa19c07af10 0x7fa19c07d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:35.275 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.274+0000 7fa1b9f27640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa19c07af10 0x7fa19c07d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.274+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fa1a809c440 con 0x7fa1b4103480 2024-09-21T01:23:35.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.275+0000 7fa1b9f27640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa19c07af10 0x7fa19c07d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0057c0 tx=0x7fa1ac0095f0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:35.280 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.279+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa1a8068dc0 con 0x7fa1b4103480 2024-09-21T01:23:35.471 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.470+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=client_mount_timeout}] v 0) -- 0x7fa1b419b2a0 con 0x7fa1b4103480 2024-09-21T01:23:35.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.474+0000 7fa1b2ffd640 1 -- 172.21.15.45:0/1849129118 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=client_mount_timeout}]=0 v20)=0 v20) ==== 119+0+0 (secure 0 0 0) 0x7fa1a806dc70 con 0x7fa1b4103480 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa19c07af10 msgr2=0x7fa19c07d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa19c07af10 0x7fa19c07d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0057c0 tx=0x7fa1ac0095f0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 msgr2=0x7fa1b4199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 0x7fa1b4199bf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa1a800b420 tx=0x7fa1a800b8e0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 shutdown_connections 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa19c07af10 0x7fa19c07d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1b4103480 0x7fa1b4199bf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 --2- 172.21.15.45:0/1849129118 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1b4102350 0x7fa1b4199580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.476+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 >> 172.21.15.45:0/1849129118 conn(0x7fa1b40fdce0 msgr2=0x7fa1b40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:35.478 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.477+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 shutdown_connections 2024-09-21T01:23:35.479 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.477+0000 7fa1bc18b640 1 -- 172.21.15.45:0/1849129118 wait complete. 2024-09-21T01:23:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 -- 172.21.15.45:0/104813300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 msgr2=0x7feba4104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 --2- 172.21.15.45:0/104813300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4104990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7feb94009880 tx=0x7feb9402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 -- 172.21.15.45:0/104813300 shutdown_connections 2024-09-21T01:23:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 --2- 172.21.15.45:0/104813300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4104990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 --2- 172.21.15.45:0/104813300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba4103450 0x7feba41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 -- 172.21.15.45:0/104813300 >> 172.21.15.45:0/104813300 conn(0x7feba40fedc0 msgr2=0x7feba4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 -- 172.21.15.45:0/104813300 shutdown_connections 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 -- 172.21.15.45:0/104813300 wait complete. 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 Processor -- start 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.921+0000 7febac6d0640 1 -- start start 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7febac6d0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba4103450 0x7feba41994b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7febac6d0640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7febac6d0640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feba410c200 con 0x7feba4104580 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7febac6d0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feba410c080 con 0x7feba4103450 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36890/0 (socket says 172.21.15.45:36890) 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 -- 172.21.15.45:0/1681893619 learned_addr learned my addr 172.21.15.45:0/1681893619 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7febaa46c640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba4103450 0x7feba41994b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:36.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 -- 172.21.15.45:0/1681893619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba4103450 msgr2=0x7feba41994b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba4103450 0x7feba41994b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 -- 172.21.15.45:0/1681893619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba419a5f0 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feba9c6b640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4199b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb94000c00 tx=0x7feb94004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.922+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7feb9403f070 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.923+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7feb94004440 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.923+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba419a880 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.923+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7feb94035030 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.923+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba419ad80 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.924+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba41038c0 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.925+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7feb94042070 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.925+0000 7feb9b7fe640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feb9007aec0 0x7feb9007d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.925+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7feb940bd8f0 con 0x7feba4104580 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.925+0000 7febaa46c640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feb9007aec0 0x7feb9007d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:36.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.926+0000 7febaa46c640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feb9007aec0 0x7feb9007d3d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feba000bbe0 tx=0x7feba000b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:36.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:35.930+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7feb9408a150 con 0x7feba4104580 2024-09-21T01:23:36.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.118+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7feba4192f80 con 0x7feba4104580 2024-09-21T01:23:36.123 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.122+0000 7feb9b7fe640 1 -- 172.21.15.45:0/1681893619 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v21)=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7feb9407fe20 con 0x7feba4104580 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feb9007aec0 msgr2=0x7feb9007d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feb9007aec0 0x7feb9007d3d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7feba000bbe0 tx=0x7feba000b390 comp rx=0 tx=0).stop 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 msgr2=0x7feba4199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4199b20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb94000c00 tx=0x7feb94004040 comp rx=0 tx=0).stop 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 shutdown_connections 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feb9007aec0 0x7feb9007d3d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feba4104580 0x7feba4199b20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.125 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 --2- 172.21.15.45:0/1681893619 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feba4103450 0x7feba41994b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:36.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 >> 172.21.15.45:0/1681893619 conn(0x7feba40fedc0 msgr2=0x7feba41079e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:36.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 shutdown_connections 2024-09-21T01:23:36.126 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:36.124+0000 7febac6d0640 1 -- 172.21.15.45:0/1681893619 wait complete. 2024-09-21T01:23:36.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:36 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1849129118' entity='client.admin' 2024-09-21T01:23:36.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:23:36.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:36 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1681893619' entity='client.admin' 2024-09-21T01:23:36.788 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:36 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1849129118' entity='client.admin' 2024-09-21T01:23:36.788 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:23:36.789 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:36 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1681893619' entity='client.admin' 2024-09-21T01:23:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 -- 172.21.15.45:0/756223528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d5006c2a0 msgr2=0x7f4d501060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 --2- 172.21.15.45:0/756223528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d5006c2a0 0x7f4d501060b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4d48009880 tx=0x7f4d4802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 -- 172.21.15.45:0/756223528 shutdown_connections 2024-09-21T01:23:37.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 --2- 172.21.15.45:0/756223528 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d50106640 0x7f4d50108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 --2- 172.21.15.45:0/756223528 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d5006c2a0 0x7f4d501060b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 -- 172.21.15.45:0/756223528 >> 172.21.15.45:0/756223528 conn(0x7f4d500fdcd0 msgr2=0x7f4d50100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:37.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 -- 172.21.15.45:0/756223528 shutdown_connections 2024-09-21T01:23:37.303 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.301+0000 7f4d57521640 1 -- 172.21.15.45:0/756223528 wait complete. 2024-09-21T01:23:37.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d57521640 1 Processor -- start 2024-09-21T01:23:37.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d57521640 1 -- start start 2024-09-21T01:23:37.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d57521640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d5006c2a0 0x7f4d50199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:37.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d57521640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 0x7f4d50199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:37.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d57521640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5010c270 con 0x7f4d50106640 2024-09-21T01:23:37.304 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d57521640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5010c0f0 con 0x7f4d5006c2a0 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d54abc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 0x7f4d50199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d54abc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 0x7f4d50199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50638/0 (socket says 172.21.15.45:50638) 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d54abc640 1 -- 172.21.15.45:0/2082740270 learned_addr learned my addr 172.21.15.45:0/2082740270 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.302+0000 7f4d552bd640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d5006c2a0 0x7f4d50199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d54abc640 1 -- 172.21.15.45:0/2082740270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d5006c2a0 msgr2=0x7f4d50199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d54abc640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d5006c2a0 0x7f4d50199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d54abc640 1 -- 172.21.15.45:0/2082740270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d5019a610 con 0x7f4d50106640 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d54abc640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 0x7f4d50199bb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400b420 tx=0x7f4d4400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4d44011630 con 0x7f4d50106640 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d5019a900 con 0x7f4d50106640 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f4d44016070 con 0x7f4d50106640 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f4d44010b80 con 0x7f4d50106640 2024-09-21T01:23:37.305 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.303+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d5019aee0 con 0x7f4d50106640 2024-09-21T01:23:37.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.304+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d50104120 con 0x7f4d50106640 2024-09-21T01:23:37.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.305+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f4d440117d0 con 0x7f4d50106640 2024-09-21T01:23:37.306 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.305+0000 7f4d4e7fc640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f4d3807aec0 0x7f4d3807d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:37.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.305+0000 7f4d552bd640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f4d3807aec0 0x7f4d3807d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:37.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.306+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f4d4409c480 con 0x7f4d50106640 2024-09-21T01:23:37.307 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.306+0000 7f4d552bd640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f4d3807aec0 0x7f4d3807d3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d48004d10 tx=0x7f4d48005d50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:37.311 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.310+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f4d44068d80 con 0x7f4d50106640 2024-09-21T01:23:37.339 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:37.339 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:37.339 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:37.340 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:37.515 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.513+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_client}] v 0) -- 0x7f4d5019b250 con 0x7f4d50106640 2024-09-21T01:23:37.519 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.518+0000 7f4d4e7fc640 1 -- 172.21.15.45:0/2082740270 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_client}]=0 v22)=0 v22) ==== 111+0+0 (secure 0 0 0) 0x7f4d4406dc30 con 0x7f4d50106640 2024-09-21T01:23:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f4d3807aec0 msgr2=0x7f4d3807d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f4d3807aec0 0x7f4d3807d3d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d48004d10 tx=0x7f4d48005d50 comp rx=0 tx=0).stop 2024-09-21T01:23:37.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 msgr2=0x7f4d50199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 0x7f4d50199bb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d4400b420 tx=0x7f4d4400b8e0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 shutdown_connections 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f4d3807aec0 0x7f4d3807d3d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d50106640 0x7f4d50199bb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 --2- 172.21.15.45:0/2082740270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4d5006c2a0 0x7f4d50199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 >> 172.21.15.45:0/2082740270 conn(0x7f4d500fdcd0 msgr2=0x7f4d50100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 shutdown_connections 2024-09-21T01:23:37.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:37.521+0000 7f4d57521640 1 -- 172.21.15.45:0/2082740270 wait complete. 2024-09-21T01:23:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:37 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:23:37.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:37 smithi090 ceph-mon[35142]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:37.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:37.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:37 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:23:37.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:37 smithi045 ceph-mon[28361]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:37.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:38.170 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 -- 172.21.15.45:0/2936528150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 msgr2=0x7fe4e4104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/2936528150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4104990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009880 tx=0x7fe4d802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 -- 172.21.15.45:0/2936528150 shutdown_connections 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/2936528150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4104990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/2936528150 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e4103450 0x7fe4e41038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 -- 172.21.15.45:0/2936528150 >> 172.21.15.45:0/2936528150 conn(0x7fe4e40fedc0 msgr2=0x7fe4e4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 -- 172.21.15.45:0/2936528150 shutdown_connections 2024-09-21T01:23:38.261 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.259+0000 7fe4e9eed640 1 -- 172.21.15.45:0/2936528150 wait complete. 2024-09-21T01:23:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.260+0000 7fe4e9eed640 1 Processor -- start 2024-09-21T01:23:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e9eed640 1 -- start start 2024-09-21T01:23:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e9eed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e4103450 0x7fe4e4199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e9eed640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:38.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e9eed640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e410c220 con 0x7fe4e4104580 2024-09-21T01:23:38.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e9eed640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e410c0a0 con 0x7fe4e4103450 2024-09-21T01:23:38.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:38.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50648/0 (socket says 172.21.15.45:50648) 2024-09-21T01:23:38.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e2ffd640 1 -- 172.21.15.45:0/3969131753 learned_addr learned my addr 172.21.15.45:0/3969131753 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:38.263 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e37fe640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e4103450 0x7fe4e4199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e2ffd640 1 -- 172.21.15.45:0/3969131753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e4103450 msgr2=0x7fe4e4199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e2ffd640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e4103450 0x7fe4e4199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.261+0000 7fe4e2ffd640 1 -- 172.21.15.45:0/3969131753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e419a560 con 0x7fe4e4104580 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.262+0000 7fe4e2ffd640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4199a90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8000c00 tx=0x7fe4d8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.262+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe4d803f070 con 0x7fe4e4104580 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.262+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe4d8004440 con 0x7fe4e4104580 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.262+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e419a7f0 con 0x7fe4e4104580 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.262+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe4d8035030 con 0x7fe4e4104580 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.262+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e419ad70 con 0x7fe4e4104580 2024-09-21T01:23:38.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.263+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4e41038c0 con 0x7fe4e4104580 2024-09-21T01:23:38.265 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.264+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fe4d8042070 con 0x7fe4e4104580 2024-09-21T01:23:38.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.265+0000 7fe4e0ff9640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe4cc07af10 0x7fe4cc07d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:38.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.265+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fe4d80bd940 con 0x7fe4e4104580 2024-09-21T01:23:38.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.265+0000 7fe4e37fe640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe4cc07af10 0x7fe4cc07d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:38.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.265+0000 7fe4e37fe640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe4cc07af10 0x7fe4cc07d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4005fd0 tx=0x7fe4d400b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:38.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.270+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe4d808a1a0 con 0x7fe4e4104580 2024-09-21T01:23:38.461 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.460+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=rados_mon_op_timeout}] v 0) -- 0x7fe4e419b0e0 con 0x7fe4e4104580 2024-09-21T01:23:38.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.464+0000 7fe4e0ff9640 1 -- 172.21.15.45:0/3969131753 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=rados_mon_op_timeout}]=0 v23)=0 v23) ==== 120+0+0 (secure 0 0 0) 0x7fe4d808f050 con 0x7fe4e4104580 2024-09-21T01:23:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe4cc07af10 msgr2=0x7fe4cc07d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe4cc07af10 0x7fe4cc07d420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4005fd0 tx=0x7fe4d400b390 comp rx=0 tx=0).stop 2024-09-21T01:23:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 msgr2=0x7fe4e4199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4199a90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8000c00 tx=0x7fe4d8004040 comp rx=0 tx=0).stop 2024-09-21T01:23:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 shutdown_connections 2024-09-21T01:23:38.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe4cc07af10 0x7fe4cc07d420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4e4104580 0x7fe4e4199a90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 --2- 172.21.15.45:0/3969131753 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe4e4103450 0x7fe4e4199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 >> 172.21.15.45:0/3969131753 conn(0x7fe4e40fedc0 msgr2=0x7fe4e41078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 shutdown_connections 2024-09-21T01:23:38.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:38.466+0000 7fe4e9eed640 1 -- 172.21.15.45:0/3969131753 wait complete. 2024-09-21T01:23:38.665 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:38 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2082740270' entity='client.admin' 2024-09-21T01:23:38.666 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:23:38.666 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:38 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3969131753' entity='client.admin' 2024-09-21T01:23:38.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:38 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2082740270' entity='client.admin' 2024-09-21T01:23:38.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:23:38.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:38 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3969131753' entity='client.admin' 2024-09-21T01:23:39.303 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:39.664 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:39 smithi045 ceph-mon[28361]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:39.664 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:39 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:23:39.693 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'client', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi045') 2024-09-21T01:23:39.693 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds', '20'], check_status=True, hostname='smithi045') 2024-09-21T01:23:39.693 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_mds_balancer', '20'], check_status=True, hostname='smithi045') 2024-09-21T01:23:39.693 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'debug_ms', '2'], check_status=True, hostname='smithi045') 2024-09-21T01:23:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:39 smithi090 ceph-mon[35142]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:39 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:23:39.903 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 -- 172.21.15.45:0/39196298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c103460 msgr2=0x7f676c1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 --2- 172.21.15.45:0/39196298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c103460 0x7f676c1038d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f675c009880 tx=0x7f675c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 -- 172.21.15.45:0/39196298 shutdown_connections 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 --2- 172.21.15.45:0/39196298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676c104590 0x7f676c1049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 --2- 172.21.15.45:0/39196298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c103460 0x7f676c1038d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 -- 172.21.15.45:0/39196298 >> 172.21.15.45:0/39196298 conn(0x7f676c0fedd0 msgr2=0x7f676c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 -- 172.21.15.45:0/39196298 shutdown_connections 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.315+0000 7f6771b4e640 1 -- 172.21.15.45:0/39196298 wait complete. 2024-09-21T01:23:40.504 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f6771b4e640 1 Processor -- start 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f6771b4e640 1 -- start start 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f6771b4e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676c103460 0x7f676c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f6771b4e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 0x7f676c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f6771b4e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10c220 con 0x7f676c104590 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f6771b4e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f676c10c0a0 con 0x7f676c103460 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 0x7f676c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676affd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 0x7f676c199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50668/0 (socket says 172.21.15.45:50668) 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676affd640 1 -- 172.21.15.45:0/4011353203 learned_addr learned my addr 172.21.15.45:0/4011353203 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676b7fe640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676c103460 0x7f676c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676affd640 1 -- 172.21.15.45:0/4011353203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676c103460 msgr2=0x7f676c199430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676affd640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676c103460 0x7f676c199430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.316+0000 7f676affd640 1 -- 172.21.15.45:0/4011353203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f676c19a570 con 0x7f676c104590 2024-09-21T01:23:40.505 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.317+0000 7f676affd640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 0x7f676c199aa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f676000d6f0 tx=0x7f676000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.317+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6760017070 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.317+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f676000de00 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.317+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f676c19a860 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.317+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f67600049d0 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.317+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f676c19ae40 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.318+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676c1038d0 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.319+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f6760004cd0 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.319+0000 7f6768ff9640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f675407aec0 0x7f675407d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.319+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f676009c8a0 con 0x7f676c104590 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.319+0000 7f676b7fe640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f675407aec0 0x7f675407d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.320+0000 7f676b7fe640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f675407aec0 0x7f675407d3d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f675c005ab0 tx=0x7f675c000bc0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:40.506 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.324+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f67600691a0 con 0x7f676c104590 2024-09-21T01:23:40.531 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.530+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mds_debug_frag}] v 0) -- 0x7f676c19b1b0 con 0x7f676c104590 2024-09-21T01:23:40.534 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.533+0000 7f6768ff9640 1 -- 172.21.15.45:0/4011353203 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_frag}]=0 v24)=0 v24) ==== 115+0+0 (secure 0 0 0) 0x7f676006e050 con 0x7f676c104590 2024-09-21T01:23:40.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f675407aec0 msgr2=0x7f675407d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f675407aec0 0x7f675407d3d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f675c005ab0 tx=0x7f675c000bc0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 msgr2=0x7f676c199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 0x7f676c199aa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f676000d6f0 tx=0x7f676000dbb0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 shutdown_connections 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f675407aec0 0x7f675407d3d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f676c104590 0x7f676c199aa0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 --2- 172.21.15.45:0/4011353203 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f676c103460 0x7f676c199430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 >> 172.21.15.45:0/4011353203 conn(0x7f676c0fedd0 msgr2=0x7f676c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 shutdown_connections 2024-09-21T01:23:40.538 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.536+0000 7f6771b4e640 1 -- 172.21.15.45:0/4011353203 wait complete. 2024-09-21T01:23:40.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 -- 172.21.15.45:0/977497207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 msgr2=0x7f3530104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 --2- 172.21.15.45:0/977497207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530104990 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3524009880 tx=0x7f352402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 -- 172.21.15.45:0/977497207 shutdown_connections 2024-09-21T01:23:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 --2- 172.21.15.45:0/977497207 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530104990 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 --2- 172.21.15.45:0/977497207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3530103450 0x7f35301038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 -- 172.21.15.45:0/977497207 >> 172.21.15.45:0/977497207 conn(0x7f35300fedc0 msgr2=0x7f3530101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 -- 172.21.15.45:0/977497207 shutdown_connections 2024-09-21T01:23:40.708 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.706+0000 7f3535391640 1 -- 172.21.15.45:0/977497207 wait complete. 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f3535391640 1 Processor -- start 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f3535391640 1 -- start start 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f3535391640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3530103450 0x7f3530199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f3535391640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f3535391640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f353010c200 con 0x7f3530104580 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f3535391640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f353010c080 con 0x7f3530103450 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f352e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f352e7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50690/0 (socket says 172.21.15.45:50690) 2024-09-21T01:23:40.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.707+0000 7f352e7fc640 1 -- 172.21.15.45:0/1564058621 learned_addr learned my addr 172.21.15.45:0/1564058621 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f352effd640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3530103450 0x7f3530199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f352e7fc640 1 -- 172.21.15.45:0/1564058621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3530103450 msgr2=0x7f3530199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f352e7fc640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3530103450 0x7f3530199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f352e7fc640 1 -- 172.21.15.45:0/1564058621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353019a540 con 0x7f3530104580 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f352e7fc640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530199a70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3524000c00 tx=0x7f3524004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f352403f070 con 0x7f3530104580 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3524004440 con 0x7f3530104580 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f353019a7d0 con 0x7f3530104580 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3524035030 con 0x7f3530104580 2024-09-21T01:23:40.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.708+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f353019acd0 con 0x7f3530104580 2024-09-21T01:23:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.709+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f353010c950 con 0x7f3530104580 2024-09-21T01:23:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.710+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f3524042070 con 0x7f3530104580 2024-09-21T01:23:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.711+0000 7f3517fff640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f351807af10 0x7f351807d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:40.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.711+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f35240bd850 con 0x7f3530104580 2024-09-21T01:23:40.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.711+0000 7f352effd640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f351807af10 0x7f351807d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:40.713 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.711+0000 7f352effd640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f351807af10 0x7f351807d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3520005d00 tx=0x7f352000b410 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:40.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.715+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f352408a130 con 0x7f3530104580 2024-09-21T01:23:40.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.904+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_op_complaint_time}] v 0) -- 0x7f35301038c0 con 0x7f3530104580 2024-09-21T01:23:40.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.908+0000 7f3517fff640 1 -- 172.21.15.45:0/1564058621 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_op_complaint_time}]=0 v25)=0 v25) ==== 121+0+0 (secure 0 0 0) 0x7f352408efe0 con 0x7f3530104580 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.910+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f351807af10 msgr2=0x7f351807d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.910+0000 7f3535391640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f351807af10 0x7f351807d420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3520005d00 tx=0x7f352000b410 comp rx=0 tx=0).stop 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.910+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 msgr2=0x7f3530199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.910+0000 7f3535391640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530199a70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3524000c00 tx=0x7f3524004040 comp rx=0 tx=0).stop 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 shutdown_connections 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f351807af10 0x7f351807d420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3530104580 0x7f3530199a70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 --2- 172.21.15.45:0/1564058621 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3530103450 0x7f3530199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 >> 172.21.15.45:0/1564058621 conn(0x7f35300fedc0 msgr2=0x7f35301078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 shutdown_connections 2024-09-21T01:23:40.913 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:40.911+0000 7f3535391640 1 -- 172.21.15.45:0/1564058621 wait complete. 2024-09-21T01:23:41.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:41 smithi045 ceph-mon[28361]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:41.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4011353203' entity='client.admin' 2024-09-21T01:23:41.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1564058621' entity='client.admin' 2024-09-21T01:23:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:41 smithi090 ceph-mon[35142]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:41.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:41 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4011353203' entity='client.admin' 2024-09-21T01:23:41.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:41 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1564058621' entity='client.admin' 2024-09-21T01:23:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 -- 172.21.15.45:0/1144462909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c103470 msgr2=0x7f7d1c103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1144462909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c103470 0x7f7d1c103880 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7d10009880 tx=0x7f7d1002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.982 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 -- 172.21.15.45:0/1144462909 shutdown_connections 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1144462909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c103470 0x7f7d1c103880 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1144462909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c102340 0x7f7d1c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 -- 172.21.15.45:0/1144462909 >> 172.21.15.45:0/1144462909 conn(0x7f7d1c0fdcb0 msgr2=0x7f7d1c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 -- 172.21.15.45:0/1144462909 shutdown_connections 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.244+0000 7f7d2109b640 1 -- 172.21.15.45:0/1144462909 wait complete. 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d2109b640 1 Processor -- start 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d2109b640 1 -- start start 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d2109b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 0x7f7d1c199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d2109b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c103470 0x7f7d1c199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d2109b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c10b110 con 0x7f7d1c102340 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d2109b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1c10af90 con 0x7f7d1c103470 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.245+0000 7f7d1ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 0x7f7d1c199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:42.983 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 0x7f7d1c199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50718/0 (socket says 172.21.15.45:50718) 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1ad9d640 1 -- 172.21.15.45:0/1682663924 learned_addr learned my addr 172.21.15.45:0/1682663924 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1a59c640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c103470 0x7f7d1c199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1ad9d640 1 -- 172.21.15.45:0/1682663924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c103470 msgr2=0x7f7d1c199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1ad9d640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c103470 0x7f7d1c199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1ad9d640 1 -- 172.21.15.45:0/1682663924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c19a570 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1a59c640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c103470 0x7f7d1c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d1ad9d640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 0x7f7d1c199430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00c8f0 tx=0x7f7d0c00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7d0c0079b0 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7d0c005020 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d1c19a860 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7d0c004c50 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.246+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d1c19ae40 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.247+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d1c1027b0 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.248+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f7d0c007b50 con 0x7f7d1c102340 2024-09-21T01:23:42.984 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.249+0000 7f7d03fff640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f7d0407af10 0x7f7d0407d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.249+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f7d0c09cb80 con 0x7f7d1c102340 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.249+0000 7f7d1a59c640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f7d0407af10 0x7f7d0407d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.249+0000 7f7d1a59c640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f7d0407af10 0x7f7d0407d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d10005f70 tx=0x7f7d10004cd0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.253+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f7d0c014030 con 0x7f7d1c102340 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.444+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mds_verify_scatter}] v 0) -- 0x7f7d1c19b150 con 0x7f7d1c102340 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.447+0000 7f7d03fff640 1 -- 172.21.15.45:0/1682663924 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_verify_scatter}]=0 v26)=0 v26) ==== 119+0+0 (secure 0 0 0) 0x7f7d0c0694a0 con 0x7f7d1c102340 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f7d0407af10 msgr2=0x7f7d0407d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f7d0407af10 0x7f7d0407d420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d10005f70 tx=0x7f7d10004cd0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 msgr2=0x7f7d1c199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 0x7f7d1c199430 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00c8f0 tx=0x7f7d0c00cdb0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 shutdown_connections 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f7d0407af10 0x7f7d0407d420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7d1c103470 0x7f7d1c199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 --2- 172.21.15.45:0/1682663924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7d1c102340 0x7f7d1c199430 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 >> 172.21.15.45:0/1682663924 conn(0x7f7d1c0fdcb0 msgr2=0x7f7d1c0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:42.985 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 shutdown_connections 2024-09-21T01:23:42.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:42.450+0000 7f7d2109b640 1 -- 172.21.15.45:0/1682663924 wait complete. 2024-09-21T01:23:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.345+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1904566289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c20106620 msgr2=0x7f2c20108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.345+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1904566289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c20106620 0x7f2c20108b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c009880 tx=0x7f2c0c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.345+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1904566289 shutdown_connections 2024-09-21T01:23:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.345+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1904566289 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c20106620 0x7f2c20108b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.345+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1904566289 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c2006c2b0 0x7f2c20106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.345+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1904566289 >> 172.21.15.45:0/1904566289 conn(0x7f2c200fdcb0 msgr2=0x7f2c20100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:43.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1904566289 shutdown_connections 2024-09-21T01:23:43.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1904566289 wait complete. 2024-09-21T01:23:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 Processor -- start 2024-09-21T01:23:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 -- start start 2024-09-21T01:23:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 0x7f2c201993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c20106620 0x7f2c20199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2010c1e0 con 0x7f2c2006c2b0 2024-09-21T01:23:43.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.346+0000 7f2c26f1e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2c2010c060 con 0x7f2c20106620 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 0x7f2c201993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 0x7f2c201993f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50734/0 (socket says 172.21.15.45:50734) 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 -- 172.21.15.45:0/1998089920 learned_addr learned my addr 172.21.15.45:0/1998089920 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 -- 172.21.15.45:0/1998089920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c20106620 msgr2=0x7f2c20199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c20106620 0x7f2c20199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 -- 172.21.15.45:0/1998089920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c2019a530 con 0x7f2c2006c2b0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c24cba640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 0x7f2c201993f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400d8d0 tx=0x7f2c1400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2c14017030 con 0x7f2c2006c2b0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2c14004020 con 0x7f2c2006c2b0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c2019a820 con 0x7f2c2006c2b0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2c14005510 con 0x7f2c2006c2b0 2024-09-21T01:23:43.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.347+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c2019ad80 con 0x7f2c2006c2b0 2024-09-21T01:23:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.348+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c20104080 con 0x7f2c2006c2b0 2024-09-21T01:23:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.349+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f2c14009a30 con 0x7f2c2006c2b0 2024-09-21T01:23:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.350+0000 7f2c1dffb640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2c1007af10 0x7f2c1007d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.350+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2c14018070 con 0x7f2c2006c2b0 2024-09-21T01:23:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.350+0000 7f2c1ffff640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2c1007af10 0x7f2c1007d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:43.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.350+0000 7f2c1ffff640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2c1007af10 0x7f2c1007d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c02a9c0 tx=0x7f2c0c004cd0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:43.355 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.354+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2c14069360 con 0x7f2c2006c2b0 2024-09-21T01:23:43.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.546+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f2c2019f780 con 0x7f2c2006c2b0 2024-09-21T01:23:43.547 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:43 smithi045 ceph-mon[28361]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:43.547 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:43 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1682663924' entity='client.admin' 2024-09-21T01:23:43.547 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.546+0000 7f2c1dffb640 1 -- 172.21.15.45:0/1998089920 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v26)=0 v26) ==== 106+0+0 (secure 0 0 0) 0x7f2c1406e210 con 0x7f2c2006c2b0 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2c1007af10 msgr2=0x7f2c1007d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2c1007af10 0x7f2c1007d420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c02a9c0 tx=0x7f2c0c004cd0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 msgr2=0x7f2c201993f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 0x7f2c201993f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2c1400d8d0 tx=0x7f2c1400dd90 comp rx=0 tx=0).stop 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 shutdown_connections 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2c1007af10 0x7f2c1007d420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2c20106620 0x7f2c20199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.548+0000 7f2c26f1e640 1 --2- 172.21.15.45:0/1998089920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2c2006c2b0 0x7f2c201993f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.549+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 >> 172.21.15.45:0/1998089920 conn(0x7f2c200fdcb0 msgr2=0x7f2c20100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:43.550 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.549+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 shutdown_connections 2024-09-21T01:23:43.551 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:43.549+0000 7f2c26f1e640 1 -- 172.21.15.45:0/1998089920 wait complete. 2024-09-21T01:23:43.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:43 smithi090 ceph-mon[35142]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:43.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:43 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1682663924' entity='client.admin' 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 -- 172.21.15.45:0/3949278867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4102360 msgr2=0x7feca41027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 --2- 172.21.15.45:0/3949278867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4102360 0x7feca41027d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fec98009880 tx=0x7fec9802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 -- 172.21.15.45:0/3949278867 shutdown_connections 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 --2- 172.21.15.45:0/3949278867 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feca4103490 0x7feca41038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 --2- 172.21.15.45:0/3949278867 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4102360 0x7feca41027d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 -- 172.21.15.45:0/3949278867 >> 172.21.15.45:0/3949278867 conn(0x7feca40fdcd0 msgr2=0x7feca4100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 -- 172.21.15.45:0/3949278867 shutdown_connections 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.190+0000 7fecab10f640 1 -- 172.21.15.45:0/3949278867 wait complete. 2024-09-21T01:23:44.206 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.191+0000 7fecab10f640 1 Processor -- start 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.191+0000 7fecab10f640 1 -- start start 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.191+0000 7fecab10f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feca4102360 0x7feca406bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.191+0000 7fecab10f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 0x7feca419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.191+0000 7fecab10f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feca410b120 con 0x7feca4103490 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.191+0000 7fecab10f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feca410afa0 con 0x7feca4102360 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 0x7feca419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 0x7feca419e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50744/0 (socket says 172.21.15.45:50744) 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 -- 172.21.15.45:0/3062123204 learned_addr learned my addr 172.21.15.45:0/3062123204 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca8eab640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feca4102360 0x7feca406bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 -- 172.21.15.45:0/3062123204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feca4102360 msgr2=0x7feca406bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feca4102360 0x7feca406bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 -- 172.21.15.45:0/3062123204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feca419e6c0 con 0x7feca4103490 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca3fff640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 0x7feca419e180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fec9000e970 tx=0x7fec9000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:44.207 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fec9000cac0 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec900054b0 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feca419e8f0 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.192+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fec90004c40 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.193+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feca419eeb0 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.194+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feca41027d0 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.194+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fec90004ef0 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.195+0000 7feca1ffb640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec9407af10 0x7fec9407d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.195+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fec9009d270 con 0x7feca4103490 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.195+0000 7feca8eab640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec9407af10 0x7fec9407d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.195+0000 7feca8eab640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec9407af10 0x7fec9407d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fec98000c00 tx=0x7fec98004cd0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:44.208 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.199+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fec90069b50 con 0x7feca4103490 2024-09-21T01:23:44.393 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.391+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=osd_op_complaint_time}] v 0) -- 0x7feca419f1d0 con 0x7feca4103490 2024-09-21T01:23:44.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.395+0000 7feca1ffb640 1 -- 172.21.15.45:0/3062123204 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_op_complaint_time}]=0 v27)=0 v27) ==== 121+0+0 (secure 0 0 0) 0x7fec9006ea00 con 0x7feca4103490 2024-09-21T01:23:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec9407af10 msgr2=0x7fec9407d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.399 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec9407af10 0x7fec9407d420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fec98000c00 tx=0x7fec98004cd0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 msgr2=0x7feca419e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 0x7feca419e180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fec9000e970 tx=0x7fec9000ed70 comp rx=0 tx=0).stop 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 shutdown_connections 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec9407af10 0x7fec9407d420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feca4103490 0x7feca419e180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 --2- 172.21.15.45:0/3062123204 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feca4102360 0x7feca406bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 >> 172.21.15.45:0/3062123204 conn(0x7feca40fdcd0 msgr2=0x7feca41067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 shutdown_connections 2024-09-21T01:23:44.400 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.398+0000 7fecab10f640 1 -- 172.21.15.45:0/3062123204 wait complete. 2024-09-21T01:23:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 -- 172.21.15.45:0/722440686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730102350 msgr2=0x7fe7301027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.412 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 --2- 172.21.15.45:0/722440686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730102350 0x7fe7301027c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe720009880 tx=0x7fe72002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 -- 172.21.15.45:0/722440686 shutdown_connections 2024-09-21T01:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 --2- 172.21.15.45:0/722440686 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730103480 0x7fe730103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 --2- 172.21.15.45:0/722440686 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730102350 0x7fe7301027c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 -- 172.21.15.45:0/722440686 >> 172.21.15.45:0/722440686 conn(0x7fe7300fdce0 msgr2=0x7fe730100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 -- 172.21.15.45:0/722440686 shutdown_connections 2024-09-21T01:23:44.413 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.411+0000 7fe735c7a640 1 -- 172.21.15.45:0/722440686 wait complete. 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.412+0000 7fe735c7a640 1 Processor -- start 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.412+0000 7fe735c7a640 1 -- start start 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.412+0000 7fe735c7a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730102350 0x7fe730199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.412+0000 7fe735c7a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 0x7fe730199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.412+0000 7fe735c7a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe73010b110 con 0x7fe730103480 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.412+0000 7fe735c7a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe73010af90 con 0x7fe730102350 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 0x7fe730199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 0x7fe730199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50754/0 (socket says 172.21.15.45:50754) 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 -- 172.21.15.45:0/369992 learned_addr learned my addr 172.21.15.45:0/369992 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72f7fe640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730102350 0x7fe730199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 -- 172.21.15.45:0/369992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730102350 msgr2=0x7fe730199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730102350 0x7fe730199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 -- 172.21.15.45:0/369992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe73019a5c0 con 0x7fe730103480 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72f7fe640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730102350 0x7fe730199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72effd640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 0x7fe730199af0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe72400d6f0 tx=0x7fe72400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe724017070 con 0x7fe730103480 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe73019a8b0 con 0x7fe730103480 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.413+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe72400de00 con 0x7fe730103480 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.414+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe7240049d0 con 0x7fe730103480 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.414+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe73019ae90 con 0x7fe730103480 2024-09-21T01:23:44.416 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.415+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe73010b860 con 0x7fe730103480 2024-09-21T01:23:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.416+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fe724004cd0 con 0x7fe730103480 2024-09-21T01:23:44.417 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.417+0000 7fe72cff9640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe71807aec0 0x7fe71807d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:44.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.417+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fe72409c990 con 0x7fe730103480 2024-09-21T01:23:44.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.417+0000 7fe72f7fe640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe71807aec0 0x7fe71807d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:44.418 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.417+0000 7fe72f7fe640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe71807aec0 0x7fe71807d3d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe720005f70 tx=0x7fe720004d70 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:44.423 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.422+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe724069290 con 0x7fe730103480 2024-09-21T01:23:44.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.609+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_mgr}] v 0) -- 0x7fe7301027c0 con 0x7fe730103480 2024-09-21T01:23:44.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.609+0000 7fe72cff9640 1 -- 172.21.15.45:0/369992 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mgr}]=0 v27)=0 v27) ==== 108+0+0 (secure 0 0 0) 0x7fe72406e140 con 0x7fe730103480 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe71807aec0 msgr2=0x7fe71807d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe71807aec0 0x7fe71807d3d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe720005f70 tx=0x7fe720004d70 comp rx=0 tx=0).stop 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 msgr2=0x7fe730199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 0x7fe730199af0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe72400d6f0 tx=0x7fe72400dbb0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 shutdown_connections 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe71807aec0 0x7fe71807d3d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe730103480 0x7fe730199af0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 --2- 172.21.15.45:0/369992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe730102350 0x7fe730199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 >> 172.21.15.45:0/369992 conn(0x7fe7300fdce0 msgr2=0x7fe7300ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 shutdown_connections 2024-09-21T01:23:44.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:44.612+0000 7fe735c7a640 1 -- 172.21.15.45:0/369992 wait complete. 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: Reconfiguring crash.smithi045 (monmap changed)... 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]} : dispatch 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: Reconfiguring daemon crash.smithi045 on smithi045 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:44.953 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:44 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3062123204' entity='client.admin' 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: Reconfiguring crash.smithi045 (monmap changed)... 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]} : dispatch 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: Reconfiguring daemon crash.smithi045 on smithi045 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:44 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3062123204' entity='client.admin' 2024-09-21T01:23:46.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:46.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:46.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:46 smithi045 ceph-mon[28361]: Reconfiguring alertmanager.smithi045 (dependencies changed)... 2024-09-21T01:23:46.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:46 smithi045 ceph-mon[28361]: Reconfiguring daemon alertmanager.smithi045 on smithi045 2024-09-21T01:23:46.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:46 smithi090 ceph-mon[35142]: Reconfiguring alertmanager.smithi045 (dependencies changed)... 2024-09-21T01:23:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:46 smithi090 ceph-mon[35142]: Reconfiguring daemon alertmanager.smithi045 on smithi045 2024-09-21T01:23:47.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.059+0000 7f2ccdea4640 1 -- 172.21.15.45:0/3122973444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 msgr2=0x7f2cc81038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:47.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.059+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/3122973444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 0x7f2cc81038c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009c20 tx=0x7f2cb802a970 comp rx=0 tx=0).stop 2024-09-21T01:23:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.059+0000 7f2ccdea4640 1 -- 172.21.15.45:0/3122973444 shutdown_connections 2024-09-21T01:23:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.059+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/3122973444 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.060+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/3122973444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 0x7f2cc81038c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.060+0000 7f2ccdea4640 1 -- 172.21.15.45:0/3122973444 >> 172.21.15.45:0/3122973444 conn(0x7f2cc80fedc0 msgr2=0x7f2cc8101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.060+0000 7f2ccdea4640 1 -- 172.21.15.45:0/3122973444 shutdown_connections 2024-09-21T01:23:47.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.060+0000 7f2ccdea4640 1 -- 172.21.15.45:0/3122973444 wait complete. 2024-09-21T01:23:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.061+0000 7f2ccdea4640 1 Processor -- start 2024-09-21T01:23:47.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.061+0000 7f2ccdea4640 1 -- start start 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.061+0000 7f2ccdea4640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 0x7f2cc81994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.061+0000 7f2ccdea4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.061+0000 7f2ccdea4640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810c1f0 con 0x7f2cc8103450 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.061+0000 7f2ccdea4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810c070 con 0x7f2cc8104580 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.45:57032/0 (socket says 172.21.15.45:57032) 2024-09-21T01:23:47.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 -- 172.21.15.45:0/152289452 learned_addr learned my addr 172.21.15.45:0/152289452 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 -- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 msgr2=0x7f2cc81994a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 0x7f2cc81994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 -- 172.21.15.45:0/152289452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc819a5e0 con 0x7f2cc8104580 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.062+0000 7f2cc6ffd640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8199b10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00d8d0 tx=0x7f2cbc00dd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.063+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2cbc017030 con 0x7f2cc8104580 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.063+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2cbc004020 con 0x7f2cc8104580 2024-09-21T01:23:47.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.063+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2cbc005510 con 0x7f2cc8104580 2024-09-21T01:23:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.063+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc819a8d0 con 0x7f2cc8104580 2024-09-21T01:23:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.063+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc819ae30 con 0x7f2cc8104580 2024-09-21T01:23:47.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.064+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cc81038c0 con 0x7f2cc8104580 2024-09-21T01:23:47.066 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.065+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f2cbc0047a0 con 0x7f2cc8104580 2024-09-21T01:23:47.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.066+0000 7f2cc4ff9640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2cb007af50 0x7f2cb007d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:47.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.066+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f2cbc018070 con 0x7f2cc8104580 2024-09-21T01:23:47.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.066+0000 7f2cc77fe640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2cb007af50 0x7f2cb007d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:47.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.066+0000 7f2cc77fe640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2cb007af50 0x7f2cb007d460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cb80094e0 tx=0x7f2cb8000f40 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:47.072 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.071+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f2cbc069350 con 0x7f2cc8104580 2024-09-21T01:23:47.262 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.260+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mds_debug_scatterstat}] v 0) -- 0x7f2cc802e640 con 0x7f2cc8104580 2024-09-21T01:23:47.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.266+0000 7f2cc4ff9640 1 -- 172.21.15.45:0/152289452 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_debug_scatterstat}]=0 v28)=0 v28) ==== 122+0+0 (secure 0 0 0) 0x7f2cbc06e200 con 0x7f2cc8104580 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2cb007af50 msgr2=0x7f2cb007d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2cb007af50 0x7f2cb007d460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2cb80094e0 tx=0x7f2cb8000f40 comp rx=0 tx=0).stop 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 msgr2=0x7f2cc8199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8199b10 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00d8d0 tx=0x7f2cbc00dd90 comp rx=0 tx=0).stop 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 shutdown_connections 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2cb007af50 0x7f2cb007d460 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2cc8104580 0x7f2cc8199b10 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:47.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 --2- 172.21.15.45:0/152289452 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2cc8103450 0x7f2cc81994a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:47.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.268+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 >> 172.21.15.45:0/152289452 conn(0x7f2cc80fedc0 msgr2=0x7f2cc81079e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:47.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.269+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 shutdown_connections 2024-09-21T01:23:47.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:47.269+0000 7f2ccdea4640 1 -- 172.21.15.45:0/152289452 wait complete. 2024-09-21T01:23:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:47 smithi045 ceph-mon[28361]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:47 smithi045 ceph-mon[28361]: from='client.? ' entity='client.admin' 2024-09-21T01:23:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:47 smithi090 ceph-mon[35142]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:47 smithi090 ceph-mon[35142]: from='client.? ' entity='client.admin' 2024-09-21T01:23:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:49 smithi045 ceph-mon[28361]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:49 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3637139196' entity='client.admin' 2024-09-21T01:23:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:49 smithi090 ceph-mon[35142]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:49 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3637139196' entity='client.admin' 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 -- 172.21.15.45:0/667716471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 msgr2=0x7f8b38108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 --2- 172.21.15.45:0/667716471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38108b00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009880 tx=0x7f8b2c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 -- 172.21.15.45:0/667716471 shutdown_connections 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 --2- 172.21.15.45:0/667716471 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38108b00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 --2- 172.21.15.45:0/667716471 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b3806c2b0 0x7f8b38106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 -- 172.21.15.45:0/667716471 >> 172.21.15.45:0/667716471 conn(0x7f8b380fdcb0 msgr2=0x7f8b38100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 -- 172.21.15.45:0/667716471 shutdown_connections 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.178+0000 7f8b3e772640 1 -- 172.21.15.45:0/667716471 wait complete. 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b3e772640 1 Processor -- start 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b3e772640 1 -- start start 2024-09-21T01:23:50.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b3e772640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b3806c2b0 0x7f8b38199420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b3e772640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b3e772640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3810c200 con 0x7f8b38106620 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b3e772640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3810c080 con 0x7f8b3806c2b0 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b377fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b377fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33380/0 (socket says 172.21.15.45:33380) 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.179+0000 7f8b377fe640 1 -- 172.21.15.45:0/3637139196 learned_addr learned my addr 172.21.15.45:0/3637139196 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b37fff640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b3806c2b0 0x7f8b38199420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b377fe640 1 -- 172.21.15.45:0/3637139196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b3806c2b0 msgr2=0x7f8b38199420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b377fe640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b3806c2b0 0x7f8b38199420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b377fe640 1 -- 172.21.15.45:0/3637139196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3819a560 con 0x7f8b38106620 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b377fe640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38199a90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c000c00 tx=0x7f8b2c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8b2c03f070 con 0x7f8b38106620 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8b2c004440 con 0x7f8b38106620 2024-09-21T01:23:50.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3819a7f0 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8b2c035030 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.180+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b3819ad70 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.181+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b3810c950 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.182+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f8b2c042070 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.182+0000 7f8b357fa640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8b2007af10 0x7f8b2007d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.182+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f8b2c0bd860 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.183+0000 7f8b37fff640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8b2007af10 0x7f8b2007d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.183+0000 7f8b37fff640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8b2007af10 0x7f8b2007d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b28005fd0 tx=0x7f8b28005e20 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.187+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8b2c08a1e0 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.377+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=debug_ms}] v 0) -- 0x7f8b38104080 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.380+0000 7f8b357fa640 1 -- 172.21.15.45:0/3637139196 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_ms}]=0 v29)=0 v29) ==== 106+0+0 (secure 0 0 0) 0x7f8b2c08f090 con 0x7f8b38106620 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.382+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8b2007af10 msgr2=0x7f8b2007d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.382+0000 7f8b3e772640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8b2007af10 0x7f8b2007d420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8b28005fd0 tx=0x7f8b28005e20 comp rx=0 tx=0).stop 2024-09-21T01:23:50.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.382+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 msgr2=0x7f8b38199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.382+0000 7f8b3e772640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38199a90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c000c00 tx=0x7f8b2c004040 comp rx=0 tx=0).stop 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 shutdown_connections 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8b2007af10 0x7f8b2007d420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8b38106620 0x7f8b38199a90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 --2- 172.21.15.45:0/3637139196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8b3806c2b0 0x7f8b38199420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 >> 172.21.15.45:0/3637139196 conn(0x7f8b380fdcb0 msgr2=0x7f8b38100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 shutdown_connections 2024-09-21T01:23:50.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:49.383+0000 7f8b3e772640 1 -- 172.21.15.45:0/3637139196 wait complete. 2024-09-21T01:23:50.850 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_frag', 'True'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.850 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_debug_scatterstat', 'True'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_op_complaint_time', '180'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'mds_verify_scatter', 'True'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_mon_op_timeout', '900'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mds', 'rados_osd_op_timeout', '900'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'client_mount_timeout', '30'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_client', '20'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.851 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_mgr', '20'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.852 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'debug_ms', '1'], check_status=True, hostname='smithi045') 2024-09-21T01:23:50.852 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mgr', 'mon_warn_on_pool_no_app', 'False'], check_status=True, hostname='smithi045') 2024-09-21T01:23:51.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 -- 172.21.15.45:0/4161962526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 msgr2=0x7fbf4c1038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/4161962526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c1038c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c009c20 tx=0x7fbf3c02a970 comp rx=0 tx=0).stop 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 -- 172.21.15.45:0/4161962526 shutdown_connections 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/4161962526 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf4c104580 0x7fbf4c104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/4161962526 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c1038c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 -- 172.21.15.45:0/4161962526 >> 172.21.15.45:0/4161962526 conn(0x7fbf4c0fedc0 msgr2=0x7fbf4c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 -- 172.21.15.45:0/4161962526 shutdown_connections 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 -- 172.21.15.45:0/4161962526 wait complete. 2024-09-21T01:23:51.034 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.032+0000 7fbf51e2a640 1 Processor -- start 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf51e2a640 1 -- start start 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf51e2a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c19b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf51e2a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf4c104580 0x7fbf4c19bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf51e2a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c10c1f0 con 0x7fbf4c103450 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf51e2a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4c10c070 con 0x7fbf4c104580 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c19b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c19b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33404/0 (socket says 172.21.15.45:33404) 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4b7fe640 1 -- 172.21.15.45:0/2917112387 learned_addr learned my addr 172.21.15.45:0/2917112387 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4affd640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf4c104580 0x7fbf4c19bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4b7fe640 1 -- 172.21.15.45:0/2917112387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf4c104580 msgr2=0x7fbf4c19bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:51.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4b7fe640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf4c104580 0x7fbf4c19bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.033+0000 7fbf4b7fe640 1 -- 172.21.15.45:0/2917112387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4c19c7b0 con 0x7fbf4c103450 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.034+0000 7fbf4b7fe640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c19b670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c005cc0 tx=0x7fbf3c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.034+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbf3c047070 con 0x7fbf4c103450 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.034+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4c19ca40 con 0x7fbf4c103450 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.034+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fbf3c030df0 con 0x7fbf4c103450 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.034+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf4c19cf40 con 0x7fbf4c103450 2024-09-21T01:23:51.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.034+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fbf3c034070 con 0x7fbf4c103450 2024-09-21T01:23:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.035+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf4c1038c0 con 0x7fbf4c103450 2024-09-21T01:23:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.036+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fbf3c04a070 con 0x7fbf4c103450 2024-09-21T01:23:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.036+0000 7fbf48ff9640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbf3407af10 0x7fbf3407d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.036+0000 7fbf4affd640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbf3407af10 0x7fbf3407d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.037+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7fbf3c0bdb70 con 0x7fbf4c103450 2024-09-21T01:23:51.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.037+0000 7fbf4affd640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbf3407af10 0x7fbf3407d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005d00 tx=0x7fbf4000b410 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:51.042 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.041+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fbf3c03f070 con 0x7fbf4c103450 2024-09-21T01:23:51.234 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.233+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_pool_no_app}] v 0) -- 0x7fbf4c1086c0 con 0x7fbf4c103450 2024-09-21T01:23:51.238 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.237+0000 7fbf48ff9640 1 -- 172.21.15.45:0/2917112387 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_pool_no_app}]=0 v30)=0 v30) ==== 125+0+0 (secure 0 0 0) 0x7fbf3c08a3d0 con 0x7fbf4c103450 2024-09-21T01:23:51.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.239+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbf3407af10 msgr2=0x7fbf3407d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:51.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.239+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbf3407af10 0x7fbf3407d420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf40005d00 tx=0x7fbf4000b410 comp rx=0 tx=0).stop 2024-09-21T01:23:51.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.239+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 msgr2=0x7fbf4c19b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:51.241 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.239+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c19b670 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbf3c005cc0 tx=0x7fbf3c003ba0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 shutdown_connections 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fbf3407af10 0x7fbf3407d420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fbf4c104580 0x7fbf4c19bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 --2- 172.21.15.45:0/2917112387 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbf4c103450 0x7fbf4c19b670 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 >> 172.21.15.45:0/2917112387 conn(0x7fbf4c0fedc0 msgr2=0x7fbf4c105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 shutdown_connections 2024-09-21T01:23:51.242 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:51.240+0000 7fbf51e2a640 1 -- 172.21.15.45:0/2917112387 wait complete. 2024-09-21T01:23:51.737 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:51 smithi045 ceph-mon[28361]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:51.766 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:51 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2917112387' entity='client.admin' 2024-09-21T01:23:51.766 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:23:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:51 smithi090 ceph-mon[35142]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:51.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:51 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2917112387' entity='client.admin' 2024-09-21T01:23:51.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:51 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:23:52.910 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:52 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:23:52.910 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:52 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:23:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:23:53.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:53 smithi045 ceph-mon[28361]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:53.671 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'mon', 'mon_op_complaint_time', '120'], check_status=True, hostname='smithi045') 2024-09-21T01:23:53.671 DEBUG:tasks.cephadm:waiting for RemoteProcess(client=, args=['sudo', '/home/ubuntu/cephtest/cephadm', '--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '9b2fd594-77b7-11ef-baf5-efdc52797490', '--', 'ceph', 'config', 'set', 'osd', 'osd_op_complaint_time', '180'], check_status=True, hostname='smithi045') 2024-09-21T01:23:53.671 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config dump 2024-09-21T01:23:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:53 smithi090 ceph-mon[35142]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:54.583 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 -- 172.21.15.45:0/1228351294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 msgr2=0x7ff6101038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 --2- 172.21.15.45:0/1228351294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff6101038c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff604009880 tx=0x7ff60402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 -- 172.21.15.45:0/1228351294 shutdown_connections 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 --2- 172.21.15.45:0/1228351294 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610104580 0x7ff610104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 --2- 172.21.15.45:0/1228351294 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff6101038c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 -- 172.21.15.45:0/1228351294 >> 172.21.15.45:0/1228351294 conn(0x7ff6100fedc0 msgr2=0x7ff610101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 -- 172.21.15.45:0/1228351294 shutdown_connections 2024-09-21T01:23:55.086 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.083+0000 7ff616e70640 1 -- 172.21.15.45:0/1228351294 wait complete. 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff616e70640 1 Processor -- start 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff616e70640 1 -- start start 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff616e70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff610199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff616e70640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610104580 0x7ff610199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff616e70640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff61010c210 con 0x7ff610103450 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff616e70640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff61010c090 con 0x7ff610104580 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff614c0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff610199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff614c0c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff610199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33424/0 (socket says 172.21.15.45:33424) 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff614c0c640 1 -- 172.21.15.45:0/3684014140 learned_addr learned my addr 172.21.15.45:0/3684014140 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:23:55.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff60ffff640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610104580 0x7ff610199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff614c0c640 1 -- 172.21.15.45:0/3684014140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610104580 msgr2=0x7ff610199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff614c0c640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610104580 0x7ff610199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.084+0000 7ff614c0c640 1 -- 172.21.15.45:0/3684014140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff61019a570 con 0x7ff610103450 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.085+0000 7ff614c0c640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff610199430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff604009230 tx=0x7ff604005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.085+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff60403f070 con 0x7ff610103450 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.085+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6040043d0 con 0x7ff610103450 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.085+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff604035030 con 0x7ff610103450 2024-09-21T01:23:55.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.085+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff61019a800 con 0x7ff610103450 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.085+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff61019acc0 con 0x7ff610103450 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.086+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6101038c0 con 0x7ff610103450 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.087+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7ff604042070 con 0x7ff610103450 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.087+0000 7ff60dffb640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff60007aec0 0x7ff60007d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.088+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7ff6040bd950 con 0x7ff610103450 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.088+0000 7ff60ffff640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff60007aec0 0x7ff60007d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:23:55.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.088+0000 7ff60ffff640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff60007aec0 0x7ff60007d3d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005fd0 tx=0x7ff5fc005850 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:23:55.093 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.092+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff60408a1b0 con 0x7ff610103450 2024-09-21T01:23:55.292 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.291+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config dump"} v 0) -- 0x7ff61019b030 con 0x7ff610103450 2024-09-21T01:23:55.293 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.292+0000 7ff60dffb640 1 -- 172.21.15.45:0/3684014140 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config dump"}]=0 v30) ==== 59+0+18270 (secure 0 0 0) 0x7ff60408f060 con 0x7ff610103450 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:WHO MASK LEVEL OPTION VALUE RO 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global dev auth_debug true 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global basic container_image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 * 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global dev debug_asserts_on_shutdown true 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global basic log_to_file true 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global basic log_to_journald false 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global basic log_to_stderr false 2024-09-21T01:23:55.294 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_allow_pool_delete true 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_clock_drift_allowed 1.000000 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_cluster_log_to_file true 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_max_pg_per_osd 10000 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_pg_warn_max_object_skew 0.000000 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_warn_on_crush_straw_calc_version_zero false 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_warn_on_legacy_crush_tunables false 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_warn_on_osd_down_out_interval_zero false 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global dev mon_warn_on_pool_pg_num_not_power_of_two false 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced mon_warn_on_too_few_osds false 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global dev ms_die_on_bug true 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global dev ms_die_on_old_message true 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced osd_pool_default_erasure_code_profile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T01:23:55.295 INFO:teuthology.orchestra.run.smithi045.stdout:global advanced osd_pool_default_pg_autoscale_mode off 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced auth_allow_insecure_global_id_reclaim false 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced auth_mon_ticket_ttl 660.000000 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced auth_service_ticket_ttl 240.000000 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced debug_mon 20/20 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced debug_ms 1/1 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced debug_paxos 20/20 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_data_avail_warn 5 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_down_mkfs_grace 300 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_mgr_mkfs_grace 240 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_op_complaint_time 120 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon dev mon_osd_prime_pg_temp true 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_osd_reporter_subtree_level osd 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T01:23:55.296 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim false 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced mon_warn_on_insecure_global_id_reclaim_allowed false 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mon advanced public_network 172.21.0.0/20 * 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced client_mount_timeout 30 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced debug_client 20/20 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced debug_mgr 20/20 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced debug_ms 1/1 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mgr/cephadm/allow_ptrace true * 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mgr/cephadm/container_init True * 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mgr/cephadm/migration_current 7 * 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mgr/dashboard/ssl_server_port 8443 * 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mgr/orchestrator/orchestrator cephadm 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mon_reweight_min_bytes_per_osd 10 2024-09-21T01:23:55.297 INFO:teuthology.orchestra.run.smithi045.stdout:mgr advanced mon_reweight_min_pgs_per_osd 4 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:mgr dev mon_warn_on_pool_no_app false 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev bdev_debug_aio true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced debug_ms 1/1 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced debug_osd 20/20 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_misdirected_ops true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_op_order true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_pg_log_writeout true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_shutdown true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_verify_cached_snaps true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_verify_missing_on_start true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd dev osd_debug_verify_stray_on_activate true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_deep_scrub_update_digest_min_age 30 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_mclock_profile high_recovery_ops 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_memory_target_autotune true 2024-09-21T01:23:55.298 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_op_complaint_time 180.000000 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_op_queue debug_random * 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_op_queue_cut_off debug_random * 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_recover_clone_overlap true 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_recovery_max_chunk 1048576 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_scrub_load_threshold 5.000000 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_scrub_max_interval 600.000000 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:osd advanced osd_shutdown_pgref_assert true 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced debug_mds 20/20 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced debug_mds_balancer 20/20 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced debug_ms 2/2 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds dev mds_debug_frag true 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds dev mds_debug_scatterstat true 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced mds_op_complaint_time 180.000000 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds dev mds_verify_scatter true 2024-09-21T01:23:55.299 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced osd_op_complaint_time 180.000000 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced rados_mon_op_timeout 900 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:mds advanced rados_osd_op_timeout 900 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client advanced client_mount_timeout 600 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client advanced debug_client 20/20 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client advanced debug_ms 1/1 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client advanced rados_mon_op_timeout 900 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client advanced rados_osd_op_timeout 900 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client.rgw advanced rgw_cache_enabled true 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client.rgw advanced rgw_enable_ops_log true 2024-09-21T01:23:55.300 INFO:teuthology.orchestra.run.smithi045.stdout:client.rgw advanced rgw_enable_usage_log true 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff60007aec0 msgr2=0x7ff60007d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff60007aec0 0x7ff60007d3d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc005fd0 tx=0x7ff5fc005850 comp rx=0 tx=0).stop 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 msgr2=0x7ff610199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff610199430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff604009230 tx=0x7ff604005e20 comp rx=0 tx=0).stop 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 shutdown_connections 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff60007aec0 0x7ff60007d3d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff610104580 0x7ff610199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 --2- 172.21.15.45:0/3684014140 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff610103450 0x7ff610199430 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 >> 172.21.15.45:0/3684014140 conn(0x7ff6100fedc0 msgr2=0x7ff610107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:23:55.301 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 shutdown_connections 2024-09-21T01:23:55.302 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:23:55.295+0000 7ff616e70640 1 -- 172.21.15.45:0/3684014140 wait complete. 2024-09-21T01:23:55.656 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:55 smithi045 ceph-mon[28361]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:55.656 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:55 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3684014140' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T01:23:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:55 smithi090 ceph-mon[35142]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:55 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3684014140' entity='client.admin' cmd={"prefix": "config dump"} : dispatch 2024-09-21T01:23:56.521 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T01:23:56.521 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:23:56.521 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T01:23:56.538 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T01:23:56.538 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_1 2024-09-21T01:23:56.594 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T01:23:56.594 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:56.594 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-21T01:23:56.594 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:56.594 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:56.595 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-21 01:22:45.012731118 +0000 2024-09-21T01:23:56.595 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-21 01:17:07.116040826 +0000 2024-09-21T01:23:56.595 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-21 01:17:07.116040826 +0000 2024-09-21T01:23:56.595 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-21 01:17:07.116040826 +0000 2024-09-21T01:23:56.595 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T01:23:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-21T01:23:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-21T01:23:56.662 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000110763 s, 4.6 MB/s 2024-09-21T01:23:56.664 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T01:23:56.721 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_2 2024-09-21T01:23:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T01:23:56.778 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:56.778 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-21T01:23:56.779 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:56.779 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:56.779 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-21 01:22:45.012731118 +0000 2024-09-21T01:23:56.779 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-21 01:17:07.445035361 +0000 2024-09-21T01:23:56.779 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-21 01:17:07.445035361 +0000 2024-09-21T01:23:56.779 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-21 01:17:07.445035361 +0000 2024-09-21T01:23:56.779 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T01:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-21T01:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-21T01:23:56.846 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000100936 s, 5.1 MB/s 2024-09-21T01:23:56.847 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T01:23:56.909 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_3 2024-09-21T01:23:56.966 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T01:23:56.966 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:56.966 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-21T01:23:56.966 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:56.967 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:56.967 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-21 01:22:45.012731118 +0000 2024-09-21T01:23:56.967 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-21 01:17:07.772029929 +0000 2024-09-21T01:23:56.967 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-21 01:17:07.772029929 +0000 2024-09-21T01:23:56.967 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-21 01:17:07.772029929 +0000 2024-09-21T01:23:56.968 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T01:23:57.032 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-21T01:23:57.032 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-21T01:23:57.033 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 0.000109418 s, 4.7 MB/s 2024-09-21T01:23:57.034 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T01:23:57.091 DEBUG:teuthology.orchestra.run.smithi045:> stat /dev/vg_nvme/lv_4 2024-09-21T01:23:57.120 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:57.120 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:57.120 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:56 smithi045 ceph-mon[28361]: Reconfiguring grafana.smithi045 (dependencies changed)... 2024-09-21T01:23:57.120 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:56 smithi045 ceph-mon[28361]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:57.121 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:56 smithi045 ceph-mon[28361]: Reconfiguring daemon grafana.smithi045 on smithi045 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Access: 2024-09-21 01:22:45.013731102 +0000 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Modify: 2024-09-21 01:17:08.094024580 +0000 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout:Change: 2024-09-21 01:17:08.094024580 +0000 2024-09-21T01:23:57.138 INFO:teuthology.orchestra.run.smithi045.stdout: Birth: 2024-09-21 01:17:08.094024580 +0000 2024-09-21T01:23:57.139 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T01:23:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records in 2024-09-21T01:23:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:1+0 records out 2024-09-21T01:23:57.202 INFO:teuthology.orchestra.run.smithi045.stderr:512 bytes copied, 9.9969e-05 s, 5.1 MB/s 2024-09-21T01:23:57.203 DEBUG:teuthology.orchestra.run.smithi045:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T01:23:57.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:57.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:23:57.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:56 smithi090 ceph-mon[35142]: Reconfiguring grafana.smithi045 (dependencies changed)... 2024-09-21T01:23:57.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:56 smithi090 ceph-mon[35142]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:57.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:56 smithi090 ceph-mon[35142]: Reconfiguring daemon grafana.smithi045 on smithi045 2024-09-21T01:23:57.261 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:23:57.261 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T01:23:57.281 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T01:23:57.281 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-09-21T01:23:57.337 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T01:23:57.337 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:57.337 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-21T01:23:57.338 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:57.338 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:57.338 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-21 01:23:15.050986031 +0000 2024-09-21T01:23:57.338 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-21 01:17:07.119891744 +0000 2024-09-21T01:23:57.338 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-21 01:17:07.119891744 +0000 2024-09-21T01:23:57.338 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-21 01:17:07.119891744 +0000 2024-09-21T01:23:57.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T01:23:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-21T01:23:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-21T01:23:57.403 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000209975 s, 2.4 MB/s 2024-09-21T01:23:57.404 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T01:23:57.460 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-21 01:23:15.050986031 +0000 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-21 01:17:07.442887211 +0000 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-21 01:17:07.442887211 +0000 2024-09-21T01:23:57.516 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-21 01:17:07.442887211 +0000 2024-09-21T01:23:57.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T01:23:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-21T01:23:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-21T01:23:57.580 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000222476 s, 2.3 MB/s 2024-09-21T01:23:57.581 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T01:23:57.637 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-09-21T01:23:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T01:23:57.692 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:57.692 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-21T01:23:57.692 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:57.692 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:57.693 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-21 01:23:15.050986031 +0000 2024-09-21T01:23:57.693 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-21 01:17:07.797882230 +0000 2024-09-21T01:23:57.693 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-21 01:17:07.797882230 +0000 2024-09-21T01:23:57.693 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-21 01:17:07.797882230 +0000 2024-09-21T01:23:57.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T01:23:57.755 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-21T01:23:57.755 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-21T01:23:57.755 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000216503 s, 2.4 MB/s 2024-09-21T01:23:57.757 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T01:23:57.813 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-09-21T01:23:57.868 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T01:23:57.868 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T01:23:57.868 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-21T01:23:57.868 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:23:57.868 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T01:23:57.868 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-21 01:23:15.051986017 +0000 2024-09-21T01:23:57.869 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-21 01:17:08.118877726 +0000 2024-09-21T01:23:57.869 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-21 01:17:08.118877726 +0000 2024-09-21T01:23:57.869 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-21 01:17:08.118877726 +0000 2024-09-21T01:23:57.869 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T01:23:57.932 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-21T01:23:57.932 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-21T01:23:57.932 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000250103 s, 2.0 MB/s 2024-09-21T01:23:57.933 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T01:23:57.989 INFO:tasks.cephadm:Deploying osd.0 on smithi045 with /dev/vg_nvme/lv_4... 2024-09-21T01:23:57.989 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T01:23:58.817 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:23:59.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:23:59 smithi045 ceph-mon[28361]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:23:59.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:23:59 smithi090 ceph-mon[35142]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:01.249 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:01 smithi045 ceph-mon[28361]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:01.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:01 smithi090 ceph-mon[35142]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:02.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:02.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:02.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:02 smithi045 ceph-mon[28361]: Reconfiguring prometheus.smithi045 (dependencies changed)... 2024-09-21T01:24:02.498 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:02 smithi045 ceph-mon[28361]: Reconfiguring daemon prometheus.smithi045 on smithi045 2024-09-21T01:24:02.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:02 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:02.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:02 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:02.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:02 smithi090 ceph-mon[35142]: Reconfiguring prometheus.smithi045 (dependencies changed)... 2024-09-21T01:24:02.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:02 smithi090 ceph-mon[35142]: Reconfiguring daemon prometheus.smithi045 on smithi045 2024-09-21T01:24:02.767 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:24:02.795 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi045:vg_nvme/lv_4 2024-09-21T01:24:03.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:03 smithi045 ceph-mon[28361]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:03.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:03 smithi090 ceph-mon[35142]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:05.501 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:24:05.623 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:05 smithi045 ceph-mon[28361]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:05.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:05 smithi090 ceph-mon[35142]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:06.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 -- 172.21.15.45:0/1262203213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6160106620 msgr2=0x7f6160108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:06.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 --2- 172.21.15.45:0/1262203213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6160106620 0x7f6160108b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6154009880 tx=0x7f615402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:24:06.699 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 -- 172.21.15.45:0/1262203213 shutdown_connections 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 --2- 172.21.15.45:0/1262203213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6160106620 0x7f6160108b00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 --2- 172.21.15.45:0/1262203213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f616006c110 0x7f6160106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 -- 172.21.15.45:0/1262203213 >> 172.21.15.45:0/1262203213 conn(0x7f61600fdcd0 msgr2=0x7f6160100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 -- 172.21.15.45:0/1262203213 shutdown_connections 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 -- 172.21.15.45:0/1262203213 wait complete. 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.696+0000 7f616558c640 1 Processor -- start 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f616558c640 1 -- start start 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f616558c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 0x7f61601994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f616558c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6160106620 0x7f6160199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f616558c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f616010c250 con 0x7f616006c110 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f616558c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f616010c0d0 con 0x7f6160106620 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f615effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 0x7f61601994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f615effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 0x7f61601994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:38266/0 (socket says 172.21.15.45:38266) 2024-09-21T01:24:06.700 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f615effd640 1 -- 172.21.15.45:0/311887431 learned_addr learned my addr 172.21.15.45:0/311887431 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.697+0000 7f615e7fc640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6160106620 0x7f6160199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f615effd640 1 -- 172.21.15.45:0/311887431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6160106620 msgr2=0x7f6160199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f615effd640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6160106620 0x7f6160199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f615effd640 1 -- 172.21.15.45:0/311887431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f616019a630 con 0x7f616006c110 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f615effd640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 0x7f61601994f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f615000b420 tx=0x7f615000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6150011630 con 0x7f616006c110 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f616019a920 con 0x7f616006c110 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6150016070 con 0x7f616006c110 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616019af00 con 0x7f616006c110 2024-09-21T01:24:06.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.698+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6150010b80 con 0x7f616006c110 2024-09-21T01:24:06.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.699+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f616010c9a0 con 0x7f616006c110 2024-09-21T01:24:06.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.700+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f61500117d0 con 0x7f616006c110 2024-09-21T01:24:06.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.701+0000 7f6147fff640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f614807f4c0 0x7f61480819d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:24:06.702 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.701+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1370+0+0 (secure 0 0 0) 0x7f615009c230 con 0x7f616006c110 2024-09-21T01:24:06.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.701+0000 7f615e7fc640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f614807f4c0 0x7f61480819d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:24:06.703 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.701+0000 7f615e7fc640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f614807f4c0 0x7f61480819d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6154004d10 tx=0x7f6154005d50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:24:06.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.706+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6150068b30 con 0x7f616006c110 2024-09-21T01:24:06.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:06.913+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6160108600 con 0x7f614807f4c0 2024-09-21T01:24:07.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:07 smithi045 ceph-mon[28361]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:07.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:24:07.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:24:07.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:07.544 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:07 smithi090 ceph-mon[35142]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:07.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:24:07.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:24:07.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:07.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: from='client.14354 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: Reconfiguring ceph-exporter.smithi090 (monmap changed)... 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]} : dispatch 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:08.517 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:08 smithi045 ceph-mon[28361]: Reconfiguring daemon ceph-exporter.smithi090 on smithi090 2024-09-21T01:24:08.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: from='client.14354 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:08.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:08.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:08.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: Reconfiguring ceph-exporter.smithi090 (monmap changed)... 2024-09-21T01:24:08.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]} : dispatch 2024-09-21T01:24:08.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:08.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:08 smithi090 ceph-mon[35142]: Reconfiguring daemon ceph-exporter.smithi090 on smithi090 2024-09-21T01:24:09.452 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:09 smithi090 ceph-mon[35142]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:09.452 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:09 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:09.452 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:09 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:09.452 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:09 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]} : dispatch 2024-09-21T01:24:09.452 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:09 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:09 smithi045 ceph-mon[28361]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]} : dispatch 2024-09-21T01:24:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: Reconfiguring crash.smithi090 (monmap changed)... 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: Reconfiguring daemon crash.smithi090 on smithi090 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/262845685' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e20b68c0-eefc-40c0-b952-799c83b7f4d6"} : dispatch 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/262845685' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e20b68c0-eefc-40c0-b952-799c83b7f4d6"}]': finished 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mgr.smithi090.zsfvyu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:24:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:10.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:10 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1583775601' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:24:10.701 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: Reconfiguring crash.smithi090 (monmap changed)... 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: Reconfiguring daemon crash.smithi090 on smithi090 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/262845685' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e20b68c0-eefc-40c0-b952-799c83b7f4d6"} : dispatch 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/262845685' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e20b68c0-eefc-40c0-b952-799c83b7f4d6"}]': finished 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mgr.smithi090.zsfvyu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:10.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:10 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1583775601' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: Reconfiguring mgr.smithi090.zsfvyu (monmap changed)... 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: Reconfiguring daemon mgr.smithi090.zsfvyu on smithi090 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:24:11.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: Reconfiguring mgr.smithi090.zsfvyu (monmap changed)... 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: Reconfiguring daemon mgr.smithi090.zsfvyu on smithi090 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T01:24:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T01:24:11.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:12.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:12 smithi090 ceph-mon[35142]: Reconfiguring mon.smithi090 (monmap changed)... 2024-09-21T01:24:12.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:12 smithi090 ceph-mon[35142]: Reconfiguring daemon mon.smithi090 on smithi090 2024-09-21T01:24:12.880 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:12 smithi045 ceph-mon[28361]: Reconfiguring mon.smithi090 (monmap changed)... 2024-09-21T01:24:12.880 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:12 smithi045 ceph-mon[28361]: Reconfiguring daemon mon.smithi090 on smithi090 2024-09-21T01:24:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T01:24:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T01:24:13.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.672 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:24:13.672 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard get-alertmanager-api-host"} : dispatch 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"} : dispatch 2024-09-21T01:24:13.867 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi045.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard get-grafana-api-url"} : dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"} : dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi045.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard get-prometheus-api-host"} : dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"} : dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi045.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:24:13.868 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:14.739 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:14 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:24:14.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:14 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:24:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:15 smithi045 ceph-mon[28361]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:15 smithi090 ceph-mon[35142]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:17 smithi045 ceph-mon[28361]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:17 smithi090 ceph-mon[35142]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:19.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:19 smithi045 ceph-mon[28361]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:19.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:19.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:19.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:19.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:19.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:19 smithi090 ceph-mon[35142]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:20.884 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T01:24:20.884 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:20.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T01:24:20.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:21.710 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:21 smithi045 ceph-mon[28361]: Deploying daemon osd.0 on smithi045 2024-09-21T01:24:21.710 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:21 smithi045 ceph-mon[28361]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:21.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:21 smithi090 ceph-mon[35142]: Deploying daemon osd.0 on smithi045 2024-09-21T01:24:21.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:21 smithi090 ceph-mon[35142]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:22.885 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:23.664 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:23 smithi045 ceph-mon[28361]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:23.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:23 smithi090 ceph-mon[35142]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:25.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:25 smithi045 ceph-mon[28361]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:25 smithi090 ceph-mon[35142]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:26.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:26.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:26.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:26.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:26.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:26.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:27.764 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:27 smithi090 ceph-mon[35142]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:27.913 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:27 smithi045 ceph-mon[28361]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:29.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:29 smithi045 ceph-mon[28361]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:29.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:29.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:29.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:29 smithi090 ceph-mon[35142]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:29.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:29.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:30.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.462+0000 7f6147fff640 1 -- 172.21.15.45:0/311887431 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6160108600 con 0x7f614807f4c0 2024-09-21T01:24:30.462 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 0 on host 'smithi045' 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f614807f4c0 msgr2=0x7f61480819d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f614807f4c0 0x7f61480819d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6154004d10 tx=0x7f6154005d50 comp rx=0 tx=0).stop 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 msgr2=0x7f61601994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 0x7f61601994f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f615000b420 tx=0x7f615000b8e0 comp rx=0 tx=0).stop 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 shutdown_connections 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f614807f4c0 0x7f61480819d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:30.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6160106620 0x7f6160199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:30.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 --2- 172.21.15.45:0/311887431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f616006c110 0x7f61601994f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:30.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 >> 172.21.15.45:0/311887431 conn(0x7f61600fdcd0 msgr2=0x7f6160100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:24:30.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 shutdown_connections 2024-09-21T01:24:30.467 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:30.466+0000 7f616558c640 1 -- 172.21.15.45:0/311887431 wait complete. 2024-09-21T01:24:31.117 DEBUG:teuthology.orchestra.run.smithi045:osd.0> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.0.service 2024-09-21T01:24:31.119 INFO:tasks.cephadm:Deploying osd.1 on smithi045 with /dev/vg_nvme/lv_3... 2024-09-21T01:24:31.120 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T01:24:31.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:31 smithi090 ceph-mon[35142]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:31.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:31.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:31.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:24:31.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:31 smithi045 ceph-mon[28361]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:31.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:31.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:31.784 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:24:31.785 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:24:31 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0[48701]: 2024-09-21T01:24:31.639+0000 7f86338ea640 -1 osd.0 0 log_to_monitors true 2024-09-21T01:24:31.821 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:24:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:32 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:24:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:32 smithi045 ceph-mon[28361]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T01:24:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:32 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:24:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:32 smithi045 ceph-mon[28361]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:24:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:32 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:24:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:32 smithi090 ceph-mon[35142]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T01:24:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:32 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T01:24:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:32 smithi090 ceph-mon[35142]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T01:24:33.514 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0[48701]: 2024-09-21T01:24:33.467+0000 7f862f86c640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T01:24:33.515 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0[48701]: 2024-09-21T01:24:33.470+0000 7f862b66a640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:24:33.515 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-mon[28361]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:33.515 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-mon[28361]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T01:24:33.515 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-mon[28361]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T01:24:33.515 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-mon[28361]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:24:33.515 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:33 smithi090 ceph-mon[35142]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:33 smithi090 ceph-mon[35142]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T01:24:33.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:33 smithi090 ceph-mon[35142]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T01:24:33.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:33 smithi090 ceph-mon[35142]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:24:33.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:34.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:34 smithi090 ceph-mon[35142]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:24:34.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:34 smithi090 ceph-mon[35142]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T01:24:34.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:34 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:34.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:34 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:34.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:34 smithi045 ceph-mon[28361]: from='osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:24:34.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:34 smithi045 ceph-mon[28361]: osdmap e8: 1 total, 0 up, 1 in 2024-09-21T01:24:34.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:34 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:34.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:34 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:34.980 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:24:35.008 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi045:vg_nvme/lv_3 2024-09-21T01:24:35.402 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] boot 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:35.730 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:35.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: osd.0 [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] boot 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:35.941 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 -- 172.21.15.45:0/989019419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88103470 msgr2=0x7fec88103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 --2- 172.21.15.45:0/989019419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88103470 0x7fec88103880 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009c20 tx=0x7fec7c02a970 comp rx=0 tx=0).stop 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 -- 172.21.15.45:0/989019419 shutdown_connections 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 --2- 172.21.15.45:0/989019419 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88103470 0x7fec88103880 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 --2- 172.21.15.45:0/989019419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec88102340 0x7fec881027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 -- 172.21.15.45:0/989019419 >> 172.21.15.45:0/989019419 conn(0x7fec880fdcb0 msgr2=0x7fec88100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 -- 172.21.15.45:0/989019419 shutdown_connections 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.921+0000 7fec8cb60640 1 -- 172.21.15.45:0/989019419 wait complete. 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8cb60640 1 Processor -- start 2024-09-21T01:24:35.942 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8cb60640 1 -- start start 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8cb60640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 0x7fec88199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8cb60640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec88103470 0x7fec88199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8cb60640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fec8810c260 con 0x7fec88102340 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8cb60640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fec8810c0e0 con 0x7fec88103470 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 0x7fec88199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 0x7fec88199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:50330/0 (socket says 172.21.15.45:50330) 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 -- 172.21.15.45:0/1745289028 learned_addr learned my addr 172.21.15.45:0/1745289028 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:24:35.943 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec85d9b640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec88103470 0x7fec88199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 -- 172.21.15.45:0/1745289028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec88103470 msgr2=0x7fec88199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec88103470 0x7fec88199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 -- 172.21.15.45:0/1745289028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8819a670 con 0x7fec88102340 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.922+0000 7fec8659c640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 0x7fec88199530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fec7800b420 tx=0x7fec7800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.923+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fec78011630 con 0x7fec88102340 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.923+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fec78016070 con 0x7fec88102340 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.923+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec8819a960 con 0x7fec88102340 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.923+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fec78010b80 con 0x7fec88102340 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.923+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec8819af40 con 0x7fec88102340 2024-09-21T01:24:35.944 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.924+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec88102830 con 0x7fec88102340 2024-09-21T01:24:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.924+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7fec780117d0 con 0x7fec88102340 2024-09-21T01:24:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.925+0000 7fec777fe640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec7007aec0 0x7fec7007d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:24:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.925+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1982+0+0 (secure 0 0 0) 0x7fec7809c730 con 0x7fec88102340 2024-09-21T01:24:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.925+0000 7fec85d9b640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec7007aec0 0x7fec7007d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:24:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.926+0000 7fec85d9b640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec7007aec0 0x7fec7007d3d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0094e0 tx=0x7fec7c000f40 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:24:35.945 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:35.930+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fec78068dd0 con 0x7fec88102340 2024-09-21T01:24:36.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:36.138+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fec88107f80 con 0x7fec7007aec0 2024-09-21T01:24:36.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:36 smithi090 ceph-mon[35142]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T01:24:36.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:24:36.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:24:36.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:36.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:36 smithi045 ceph-mon[28361]: osdmap e10: 1 total, 1 up, 1 in 2024-09-21T01:24:36.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:24:36.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:24:36.795 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:37.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:37 smithi045 ceph-mon[28361]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:37.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:37 smithi045 ceph-mon[28361]: from='client.14372 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:37.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:37 smithi090 ceph-mon[35142]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:37.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:37 smithi090 ceph-mon[35142]: from='client.14372 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:24:37.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:39.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:39 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:39.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:39 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:40.718 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi045 2024-09-21T01:24:40.718 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi045 to 11844M 2024-09-21T01:24:40.718 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:40.718 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2275621100' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b10dd316-1566-4bff-933b-1b96b16a3378"} : dispatch 2024-09-21T01:24:40.719 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2275621100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b10dd316-1566-4bff-933b-1b96b16a3378"}]': finished 2024-09-21T01:24:40.719 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T01:24:40.719 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:40 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:24:40.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi045 2024-09-21T01:24:40.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi045 to 11844M 2024-09-21T01:24:40.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:40.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2275621100' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b10dd316-1566-4bff-933b-1b96b16a3378"} : dispatch 2024-09-21T01:24:40.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2275621100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b10dd316-1566-4bff-933b-1b96b16a3378"}]': finished 2024-09-21T01:24:40.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T01:24:40.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:40 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:24:41.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:41 smithi090 ceph-mon[35142]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:41.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:41 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3468833487' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:24:41.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:41 smithi045 ceph-mon[28361]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:41.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3468833487' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:24:42.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:42 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:42.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:42 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:42.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:42 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:42.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:42 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:43.617 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:43 smithi045 ceph-mon[28361]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:43.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:43 smithi090 ceph-mon[35142]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:45.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:45.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:45.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:45.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:45.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:45.065 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:44 smithi045 ceph-mon[28361]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:24:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:45.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:44 smithi090 ceph-mon[35142]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:47.208 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:47 smithi045 ceph-mon[28361]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:47.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:47 smithi090 ceph-mon[35142]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:48 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T01:24:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:48 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:48.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:48 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T01:24:48.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:48 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:24:49.211 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:49 smithi045 ceph-mon[28361]: Deploying daemon osd.1 on smithi045 2024-09-21T01:24:49.211 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:49 smithi045 ceph-mon[28361]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:49.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:49 smithi090 ceph-mon[35142]: Deploying daemon osd.1 on smithi045 2024-09-21T01:24:49.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:49 smithi090 ceph-mon[35142]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:51.307 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:51 smithi045 ceph-mon[28361]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:51 smithi090 ceph-mon[35142]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:53 smithi045 ceph-mon[28361]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:53.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:53 smithi090 ceph-mon[35142]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:53.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:24:54.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:54.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:54.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:54.493 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:24:54.493 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:54.493 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:55.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:55 smithi090 ceph-mon[35142]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:55.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:55 smithi045 ceph-mon[28361]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:56.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:56.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:56.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:56.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:57.916 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:57 smithi090 ceph-mon[35142]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:57.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:57 smithi045 ceph-mon[28361]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:58.067 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 1 on host 'smithi045' 2024-09-21T01:24:58.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.065+0000 7fec777fe640 1 -- 172.21.15.45:0/1745289028 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fec88107f80 con 0x7fec7007aec0 2024-09-21T01:24:58.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec7007aec0 msgr2=0x7fec7007d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:58.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec7007aec0 0x7fec7007d3d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0094e0 tx=0x7fec7c000f40 comp rx=0 tx=0).stop 2024-09-21T01:24:58.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 msgr2=0x7fec88199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:24:58.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 0x7fec88199530 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fec7800b420 tx=0x7fec7800b8e0 comp rx=0 tx=0).stop 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 shutdown_connections 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fec7007aec0 0x7fec7007d3d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fec88103470 0x7fec88199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 --2- 172.21.15.45:0/1745289028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fec88102340 0x7fec88199530 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 >> 172.21.15.45:0/1745289028 conn(0x7fec880fdcb0 msgr2=0x7fec88106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 shutdown_connections 2024-09-21T01:24:58.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:24:58.067+0000 7fec8cb60640 1 -- 172.21.15.45:0/1745289028 wait complete. 2024-09-21T01:24:59.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:59 smithi090 ceph-mon[35142]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:59.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:59.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:59.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:24:59 smithi090 ceph-mon[35142]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:24:59.266 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:59 smithi045 ceph-mon[28361]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:24:59.266 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:59.266 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:24:59.266 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:24:59 smithi045 ceph-mon[28361]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T01:24:59.296 DEBUG:teuthology.orchestra.run.smithi045:osd.1> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.1.service 2024-09-21T01:24:59.298 INFO:tasks.cephadm:Deploying osd.2 on smithi045 with /dev/vg_nvme/lv_2... 2024-09-21T01:24:59.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T01:25:00.597 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:25:00.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:00 smithi045 ceph-mon[28361]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T01:25:00.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:00 smithi045 ceph-mon[28361]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T01:25:00.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:00 smithi045 ceph-mon[28361]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:25:00.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:00 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:00.922 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:00 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1[55451]: 2024-09-21T01:25:00.640+0000 7ffacfc66640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T01:25:00.922 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:00 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1[55451]: 2024-09-21T01:25:00.643+0000 7ffacba64640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:25:00.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:00 smithi090 ceph-mon[35142]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T01:25:00.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:00 smithi090 ceph-mon[35142]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T01:25:00.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:00 smithi090 ceph-mon[35142]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:25:00.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:00 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:01.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:01 smithi045 ceph-mon[28361]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:25:01.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:01 smithi045 ceph-mon[28361]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:25:01.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:01 smithi045 ceph-mon[28361]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T01:25:01.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:01 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:01.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:01 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:01 smithi090 ceph-mon[35142]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:25:01.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:01 smithi090 ceph-mon[35142]: from='osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:25:01.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:01 smithi090 ceph-mon[35142]: osdmap e13: 2 total, 1 up, 2 in 2024-09-21T01:25:01.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:01 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:01.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:01 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:02.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:25:02.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:25:02.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:02.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] boot 2024-09-21T01:25:02.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T01:25:02.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:02.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:02.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:02 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:25:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:25:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: osd.1 [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] boot 2024-09-21T01:25:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T01:25:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T01:25:03.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:03.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:02 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:03.442 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:25:03.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi045:vg_nvme/lv_2 2024-09-21T01:25:03.976 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:25:04.058 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:03 smithi045 ceph-mon[28361]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:25:04.059 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:03 smithi045 ceph-mon[28361]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T01:25:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:03 smithi090 ceph-mon[35142]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T01:25:04.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:03 smithi090 ceph-mon[35142]: osdmap e15: 2 total, 2 up, 2 in 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 -- 172.21.15.45:0/2310317323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c102340 msgr2=0x7f335c1027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 --2- 172.21.15.45:0/2310317323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c102340 0x7f335c1027b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f334c009880 tx=0x7f334c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 -- 172.21.15.45:0/2310317323 shutdown_connections 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 --2- 172.21.15.45:0/2310317323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c103470 0x7f335c103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 --2- 172.21.15.45:0/2310317323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c102340 0x7f335c1027b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 -- 172.21.15.45:0/2310317323 >> 172.21.15.45:0/2310317323 conn(0x7f335c0fdcb0 msgr2=0x7f335c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 -- 172.21.15.45:0/2310317323 shutdown_connections 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 -- 172.21.15.45:0/2310317323 wait complete. 2024-09-21T01:25:04.775 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.769+0000 7f3362adc640 1 Processor -- start 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3362adc640 1 -- start start 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3362adc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c102340 0x7f335c199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3362adc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 0x7f335c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3362adc640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f335c10b0e0 con 0x7f335c103470 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3362adc640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f335c10af60 con 0x7f335c102340 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 0x7f335c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 0x7f335c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:41468/0 (socket says 172.21.15.45:41468) 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 -- 172.21.15.45:0/4046077844 learned_addr learned my addr 172.21.15.45:0/4046077844 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3360878640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c102340 0x7f335c199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 -- 172.21.15.45:0/4046077844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c102340 msgr2=0x7f335c199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c102340 0x7f335c199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 -- 172.21.15.45:0/4046077844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f335c19a540 con 0x7f335c103470 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f3360878640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c102340 0x7f335c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:25:04.776 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.770+0000 7f335bfff640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 0x7f335c199a70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f334400d8d0 tx=0x7f334400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.771+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3344017070 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.771+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3344010460 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.771+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f334400f5f0 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.771+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f335c19a830 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.771+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f335c19ae10 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.772+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f335c10b830 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.773+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f3344010610 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.773+0000 7f3359ffb640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f334807aec0 0x7f334807d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.773+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2414+0+0 (secure 0 0 0) 0x7f334409cf10 con 0x7f335c103470 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.773+0000 7f3360878640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f334807aec0 0x7f334807d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:04.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.774+0000 7f3360878640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f334807aec0 0x7f334807d3d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f334c004d10 tx=0x7f334c005d50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:04.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.778+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3344069400 con 0x7f335c103470 2024-09-21T01:25:04.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:04.980+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f335c06be00 con 0x7f334807aec0 2024-09-21T01:25:05.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:05 smithi045 ceph-mon[28361]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:05.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:25:05.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:25:05.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:05 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:06.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:05 smithi090 ceph-mon[35142]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:06.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:05 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:25:06.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:05 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:25:06.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:05 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='client.14390 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:06.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:07.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='client.14390 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:07.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:07.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:07.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:07 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi045 2024-09-21T01:25:07.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:07 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi045 to 5922M 2024-09-21T01:25:07.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:07 smithi045 ceph-mon[28361]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:07.887 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:07 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi045 2024-09-21T01:25:08.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:07 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi045 to 5922M 2024-09-21T01:25:08.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:07 smithi090 ceph-mon[35142]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:08.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:08.801 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:08 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/313096885' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "80cb32e4-abb7-4cd6-b2af-6e254e3c592b"} : dispatch 2024-09-21T01:25:08.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:08 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/313096885' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80cb32e4-abb7-4cd6-b2af-6e254e3c592b"}]': finished 2024-09-21T01:25:08.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:08 smithi045 ceph-mon[28361]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T01:25:08.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:08.802 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:08 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2865932825' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:25:09.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:08 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/313096885' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "80cb32e4-abb7-4cd6-b2af-6e254e3c592b"} : dispatch 2024-09-21T01:25:09.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:08 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/313096885' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80cb32e4-abb7-4cd6-b2af-6e254e3c592b"}]': finished 2024-09-21T01:25:09.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:08 smithi090 ceph-mon[35142]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T01:25:09.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:09.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:08 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2865932825' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:25:10.166 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:09 smithi045 ceph-mon[28361]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:09 smithi090 ceph-mon[35142]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:11 smithi045 ceph-mon[28361]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:11 smithi090 ceph-mon[35142]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:14.050 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:13 smithi045 ceph-mon[28361]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:13 smithi090 ceph-mon[35142]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:15 smithi045 ceph-mon[28361]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T01:25:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:16.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:15 smithi090 ceph-mon[35142]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T01:25:16.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:16.910 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:16 smithi045 ceph-mon[28361]: Deploying daemon osd.2 on smithi045 2024-09-21T01:25:17.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:16 smithi090 ceph-mon[35142]: Deploying daemon osd.2 on smithi045 2024-09-21T01:25:17.904 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:17 smithi045 ceph-mon[28361]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:17 smithi090 ceph-mon[35142]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:19.999 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:19 smithi045 ceph-mon[28361]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:19 smithi090 ceph-mon[35142]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:20.904 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:20 smithi045 ceph-mon[28361]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:21.169 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:20 smithi090 ceph-mon[35142]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:21.870 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:21.871 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:21.871 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:22.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:22.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:22 smithi045 ceph-mon[28361]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:22 smithi090 ceph-mon[35142]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:25.005 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.004+0000 7f3359ffb640 1 -- 172.21.15.45:0/4046077844 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f335c06be00 con 0x7f334807aec0 2024-09-21T01:25:25.005 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 2 on host 'smithi045' 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f334807aec0 msgr2=0x7f334807d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f334807aec0 0x7f334807d3d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f334c004d10 tx=0x7f334c005d50 comp rx=0 tx=0).stop 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 msgr2=0x7f335c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 0x7f335c199a70 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f334400d8d0 tx=0x7f334400dd90 comp rx=0 tx=0).stop 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 shutdown_connections 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f334807aec0 0x7f334807d3d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f335c103470 0x7f335c199a70 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 --2- 172.21.15.45:0/4046077844 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f335c102340 0x7f335c199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:25.007 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 >> 172.21.15.45:0/4046077844 conn(0x7f335c0fdcb0 msgr2=0x7f335c06b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:25.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 shutdown_connections 2024-09-21T01:25:25.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:25.007+0000 7f3362adc640 1 -- 172.21.15.45:0/4046077844 wait complete. 2024-09-21T01:25:25.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:25 smithi045 ceph-mon[28361]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:25.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:25 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:25.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:25 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:25.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:25 smithi090 ceph-mon[35142]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:25.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:25 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:25.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:25 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:25.796 DEBUG:teuthology.orchestra.run.smithi045:osd.2> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.2.service 2024-09-21T01:25:25.799 INFO:tasks.cephadm:Deploying osd.3 on smithi045 with /dev/vg_nvme/lv_1... 2024-09-21T01:25:25.799 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T01:25:26.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:26 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2[61746]: 2024-09-21T01:25:26.163+0000 7fbb8eb0d640 -1 osd.2 0 log_to_monitors true 2024-09-21T01:25:26.435 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:25:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:27 smithi045 ceph-mon[28361]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:27 smithi045 ceph-mon[28361]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:25:27.400 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:27 smithi090 ceph-mon[35142]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:27.400 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:27 smithi090 ceph-mon[35142]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T01:25:28.363 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-mon[28361]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T01:25:28.363 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-mon[28361]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T01:25:28.363 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-mon[28361]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:25:28.363 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:28.363 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-mon[28361]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:25:28.363 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2[61746]: 2024-09-21T01:25:28.037+0000 7fbb8aa8f640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T01:25:28.364 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:28 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2[61746]: 2024-09-21T01:25:28.040+0000 7fbb8688d640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:25:28.415 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:28 smithi090 ceph-mon[35142]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T01:25:28.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:28 smithi090 ceph-mon[35142]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T01:25:28.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:28 smithi090 ceph-mon[35142]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:25:28.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:28.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:28 smithi090 ceph-mon[35142]: from='osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:25:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:29 smithi045 ceph-mon[28361]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:29 smithi045 ceph-mon[28361]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T01:25:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:29 smithi090 ceph-mon[35142]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:29 smithi090 ceph-mon[35142]: osdmap e18: 3 total, 2 up, 3 in 2024-09-21T01:25:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:29.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:29.829 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:25:29.858 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi045:vg_nvme/lv_1 2024-09-21T01:25:30.130 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] boot 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: osd.2 [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] boot 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:30.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 -- 172.21.15.45:0/2213901420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 msgr2=0x7f7114104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 --2- 172.21.15.45:0/2213901420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114104990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7108009880 tx=0x7f710802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:25:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 -- 172.21.15.45:0/2213901420 shutdown_connections 2024-09-21T01:25:30.636 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 --2- 172.21.15.45:0/2213901420 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114104990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 --2- 172.21.15.45:0/2213901420 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7114103450 0x7f71141038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 -- 172.21.15.45:0/2213901420 >> 172.21.15.45:0/2213901420 conn(0x7f71140fedc0 msgr2=0x7f7114101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 -- 172.21.15.45:0/2213901420 shutdown_connections 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 -- 172.21.15.45:0/2213901420 wait complete. 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.635+0000 7f7119f15640 1 Processor -- start 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7119f15640 1 -- start start 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7119f15640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7114103450 0x7f7114199400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7119f15640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7119f15640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f711410c200 con 0x7f7114104580 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7119f15640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f711410c080 con 0x7f7114103450 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45872/0 (socket says 172.21.15.45:45872) 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 -- 172.21.15.45:0/2648941802 learned_addr learned my addr 172.21.15.45:0/2648941802 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:25:30.637 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f71137fe640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7114103450 0x7f7114199400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:30.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 -- 172.21.15.45:0/2648941802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7114103450 msgr2=0x7f7114199400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:30.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7114103450 0x7f7114199400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:30.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 -- 172.21.15.45:0/2648941802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f711419a540 con 0x7f7114104580 2024-09-21T01:25:30.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.636+0000 7f7112ffd640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114199a70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7108000c00 tx=0x7f7108004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.637+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f710803f070 con 0x7f7114104580 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.637+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7108004440 con 0x7f7114104580 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.637+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711419a7d0 con 0x7f7114104580 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.637+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7108035030 con 0x7f7114104580 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.637+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711419ad50 con 0x7f7114104580 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.638+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71141038c0 con 0x7f7114104580 2024-09-21T01:25:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.639+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 18) ==== 103264+0+0 (secure 0 0 0) 0x7f7108042070 con 0x7f7114104580 2024-09-21T01:25:30.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.639+0000 7f7110ff9640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f70fc07af10 0x7f70fc07d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:30.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.640+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 2846+0+0 (secure 0 0 0) 0x7f71080be3f0 con 0x7f7114104580 2024-09-21T01:25:30.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.640+0000 7f71137fe640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f70fc07af10 0x7f70fc07d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:30.640 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.640+0000 7f71137fe640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f70fc07af10 0x7f70fc07d420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7104005e10 tx=0x7f7104005da0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:30.644 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.644+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f710808a710 con 0x7f7114104580 2024-09-21T01:25:30.849 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:30.848+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f7114108fc0 con 0x7f70fc07af10 2024-09-21T01:25:31.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:31 smithi045 ceph-mon[28361]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:31.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:31 smithi045 ceph-mon[28361]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T01:25:31.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:25:31.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:25:31.149 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:25:31.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:31 smithi090 ceph-mon[35142]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T01:25:31.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:31 smithi090 ceph-mon[35142]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T01:25:31.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:25:31.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:25:31.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:25:31.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:32.143 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 ceph-mon[28361]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:32.143 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T01:25:32.143 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 ceph-mon[28361]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T01:25:32.143 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:25:32.403 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63642]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63642]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63642]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63642]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63645]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63645]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63645]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63636]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63636]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63636]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63636]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.404 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63639]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:25:32.405 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63639]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.405 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63639]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.405 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63639]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.489 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 ceph-mon[35142]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi045:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:32.489 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T01:25:32.489 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 ceph-mon[35142]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T01:25:32.489 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63658]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63658]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63658]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63658]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63661]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63661]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63661]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63661]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.670 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63649]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63649]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63649]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63649]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63655]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.671 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:25:32 smithi045 sudo[63655]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37667]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T01:25:32.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37667]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37667]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37667]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37670]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T01:25:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T01:25:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37670]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T01:25:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:32 smithi090 sudo[37670]: pam_unix(sudo:session): session closed for user root 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:25:33.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:25:33.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:25:33.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:25:33.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:25:33.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi045"} : dispatch 2024-09-21T01:25:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mon metadata", "id": "smithi090"} : dispatch 2024-09-21T01:25:33.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T01:25:33.500 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:34.080 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:34.079+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f7108081ea0 con 0x7f7114104580 2024-09-21T01:25:34.344 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:34 smithi045 ceph-mon[28361]: osdmap e23: 3 total, 3 up, 3 in 2024-09-21T01:25:34.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:34 smithi090 ceph-mon[35142]: osdmap e23: 3 total, 3 up, 3 in 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: mgrmap e19: smithi045.txjudz(active, since 2m), standbys: smithi090.zsfvyu 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi045 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:35.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi045 to 3948M 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:35.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: mgrmap e19: smithi045.txjudz(active, since 2m), standbys: smithi090.zsfvyu 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi045 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi045 to 3948M 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:35.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:35.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:35.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:35.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:35.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:36.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:36 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/216726001' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1aaec02f-d282-4505-a860-3d77532911df"} : dispatch 2024-09-21T01:25:36.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:36 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/216726001' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1aaec02f-d282-4505-a860-3d77532911df"}]': finished 2024-09-21T01:25:36.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:36 smithi045 ceph-mon[28361]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T01:25:36.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:36.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:36 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/216726001' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1aaec02f-d282-4505-a860-3d77532911df"} : dispatch 2024-09-21T01:25:36.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:36 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/216726001' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1aaec02f-d282-4505-a860-3d77532911df"}]': finished 2024-09-21T01:25:36.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:36 smithi090 ceph-mon[35142]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T01:25:36.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:37.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:37 smithi045 ceph-mon[28361]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:37.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:37 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/413607540' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:25:37.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:37 smithi090 ceph-mon[35142]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:37.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:37 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/413607540' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:25:38.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:38.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:39.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:39 smithi045 ceph-mon[28361]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:39.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:39 smithi090 ceph-mon[35142]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:41.274 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:41 smithi045 ceph-mon[28361]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:41 smithi090 ceph-mon[35142]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:43.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:43 smithi045 ceph-mon[28361]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:43.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:43 smithi090 ceph-mon[35142]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T01:25:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:44 smithi045 ceph-mon[28361]: Deploying daemon osd.3 on smithi045 2024-09-21T01:25:44.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T01:25:44.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:44.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:44 smithi090 ceph-mon[35142]: Deploying daemon osd.3 on smithi045 2024-09-21T01:25:45.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:45 smithi045 ceph-mon[28361]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:45.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:45 smithi090 ceph-mon[35142]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:47.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:47 smithi045 ceph-mon[28361]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:47.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:47 smithi090 ceph-mon[35142]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:49.413 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:49 smithi045 ceph-mon[28361]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:49.414 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:49.414 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:49.414 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:49.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:49 smithi090 ceph-mon[35142]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:49.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:25:49.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:49.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:51.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:51 smithi045 ceph-mon[28361]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:51 smithi090 ceph-mon[35142]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:52.780 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.778+0000 7f7110ff9640 1 -- 172.21.15.45:0/2648941802 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7114108fc0 con 0x7f70fc07af10 2024-09-21T01:25:52.780 INFO:teuthology.orchestra.run.smithi045.stdout:Created osd(s) 3 on host 'smithi045' 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f70fc07af10 msgr2=0x7f70fc07d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f70fc07af10 0x7f70fc07d420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7104005e10 tx=0x7f7104005da0 comp rx=0 tx=0).stop 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 msgr2=0x7f7114199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114199a70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7108000c00 tx=0x7f7108004040 comp rx=0 tx=0).stop 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 shutdown_connections 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f70fc07af10 0x7f70fc07d420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7114104580 0x7f7114199a70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 --2- 172.21.15.45:0/2648941802 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7114103450 0x7f7114199400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.780+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 >> 172.21.15.45:0/2648941802 conn(0x7f71140fedc0 msgr2=0x7f71141078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.781+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 shutdown_connections 2024-09-21T01:25:52.782 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:25:52.781+0000 7f7119f15640 1 -- 172.21.15.45:0/2648941802 wait complete. 2024-09-21T01:25:53.312 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:53 smithi045 ceph-mon[28361]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:53.312 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:53.312 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:53.312 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:53.313 DEBUG:teuthology.orchestra.run.smithi045:osd.3> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.3.service 2024-09-21T01:25:53.316 INFO:tasks.cephadm:Deploying osd.4 on smithi090 with /dev/vg_nvme/lv_4... 2024-09-21T01:25:53.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T01:25:53.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:53 smithi090 ceph-mon[35142]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:53.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:25:53.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:53.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:53.573 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:25:54.111 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:25:53 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:25:53.817+0000 7ff0a7f02640 -1 osd.3 0 log_to_monitors true 2024-09-21T01:25:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:54 smithi090 ceph-mon[35142]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T01:25:54.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:54 smithi045 ceph-mon[28361]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T01:25:55.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:55 smithi090 ceph-mon[35142]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:55.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:55 smithi090 ceph-mon[35142]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T01:25:55.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:55 smithi090 ceph-mon[35142]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T01:25:55.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:55 smithi090 ceph-mon[35142]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:25:55.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:55.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-mon[28361]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:55.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-mon[28361]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T01:25:55.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-mon[28361]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T01:25:55.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-mon[28361]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]} : dispatch 2024-09-21T01:25:55.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:55.393 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:25:55.119+0000 7ff0a3e84640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T01:25:55.393 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:25:55 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:25:55.123+0000 7ff09fc82640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:25:55.985 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:25:56.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-09-21T01:25:56.265 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:25:56.374 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:56 smithi045 ceph-mon[28361]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:25:56.374 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:56 smithi045 ceph-mon[28361]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T01:25:56.374 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:56.374 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:56.374 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:56.374 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:56 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:56.386 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:56 smithi090 ceph-mon[35142]: from='osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi045", "root=default"]}]': finished 2024-09-21T01:25:56.386 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:56 smithi090 ceph-mon[35142]: osdmap e26: 4 total, 3 up, 4 in 2024-09-21T01:25:56.386 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:56.386 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:56.386 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:56.386 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:56 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.704+0000 7f3ca84b6640 1 -- 172.21.15.90:0/3005574890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca006c2f0 msgr2=0x7f3ca01060a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.704+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/3005574890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca006c2f0 0x7f3ca01060a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c009c20 tx=0x7f3c9c02a970 comp rx=0 tx=0).stop 2024-09-21T01:25:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 -- 172.21.15.90:0/3005574890 shutdown_connections 2024-09-21T01:25:56.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/3005574890 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca0106630 0x7f3ca0108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/3005574890 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca006c2f0 0x7f3ca01060a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 -- 172.21.15.90:0/3005574890 >> 172.21.15.90:0/3005574890 conn(0x7f3ca00fdca0 msgr2=0x7f3ca0100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 -- 172.21.15.90:0/3005574890 shutdown_connections 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 -- 172.21.15.90:0/3005574890 wait complete. 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 Processor -- start 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.705+0000 7f3ca84b6640 1 -- start start 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca84b6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca006c2f0 0x7f3ca01993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca84b6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 0x7f3ca0199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca84b6640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca010c1e0 con 0x7f3ca006c2f0 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca84b6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3ca010c060 con 0x7f3ca0106630 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 0x7f3ca0199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:56.708 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca6252640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca006c2f0 0x7f3ca01993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 0x7f3ca0199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57218/0 (socket says 172.21.15.90:57218) 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 -- 172.21.15.90:0/134888843 learned_addr learned my addr 172.21.15.90:0/134888843 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 -- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca006c2f0 msgr2=0x7f3ca01993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca006c2f0 0x7f3ca01993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 -- 172.21.15.90:0/134888843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ca019a520 con 0x7f3ca0106630 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca5a51640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 0x7f3ca0199a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c810 tx=0x7f3c9000ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3c90004020 con 0x7f3ca0106630 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3c90007e40 con 0x7f3ca0106630 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ca019a810 con 0x7f3ca0106630 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3c90005630 con 0x7f3ca0106630 2024-09-21T01:25:56.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.706+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ca019adf0 con 0x7f3ca0106630 2024-09-21T01:25:56.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.707+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ca0104090 con 0x7f3ca0106630 2024-09-21T01:25:56.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.708+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f3c900041c0 con 0x7f3ca0106630 2024-09-21T01:25:56.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.709+0000 7f3c977fe640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3c8c07af30 0x7f3c8c07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:25:56.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.709+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3664+0+0 (secure 0 0 0) 0x7f3c9009d560 con 0x7f3ca0106630 2024-09-21T01:25:56.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.709+0000 7f3ca6252640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3c8c07af30 0x7f3c8c07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:25:56.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.710+0000 7f3ca6252640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3c8c07af30 0x7f3c8c07d440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c02f870 tx=0x7f3c9c03c040 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:25:56.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.714+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3c90014030 con 0x7f3ca0106630 2024-09-21T01:25:56.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:25:56.901+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3ca0108570 con 0x7f3c8c07af30 2024-09-21T01:25:57.278 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:25:57.278 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:25:57.278 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:57.279 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] boot 2024-09-21T01:25:57.279 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T01:25:57.279 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:57.279 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:25:57.279 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:25:57.279 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: osd.3 [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] boot 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:25:57.386 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:58.336 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:58 smithi090 ceph-mon[35142]: from='client.24207 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:58.336 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:58 smithi090 ceph-mon[35142]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T01:25:58.381 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:58 smithi045 ceph-mon[28361]: from='client.24207 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:25:58.381 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:58 smithi045 ceph-mon[28361]: osdmap e28: 4 total, 4 up, 4 in 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:59.395 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:59.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/272606511' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85e89409-05b7-432d-9044-3b730f4f68b8"} : dispatch 2024-09-21T01:25:59.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85e89409-05b7-432d-9044-3b730f4f68b8"} : dispatch 2024-09-21T01:25:59.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85e89409-05b7-432d-9044-3b730f4f68b8"}]': finished 2024-09-21T01:25:59.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T01:25:59.396 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:25:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/272606511' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85e89409-05b7-432d-9044-3b730f4f68b8"} : dispatch 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "85e89409-05b7-432d-9044-3b730f4f68b8"} : dispatch 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "85e89409-05b7-432d-9044-3b730f4f68b8"}]': finished 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T01:25:59.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:25:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi045 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi045 to 2961M 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:00.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:00 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/592855629' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:26:00.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi045 2024-09-21T01:26:00.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi045 to 2961M 2024-09-21T01:26:00.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:00.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:00.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:00.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:00.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:00 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/592855629' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:26:01.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:01 smithi045 ceph-mon[28361]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:01 smithi090 ceph-mon[35142]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:03.814 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:03 smithi090 ceph-mon[35142]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:03.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:03 smithi045 ceph-mon[28361]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:05.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:05 smithi090 ceph-mon[35142]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:05.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:05 smithi045 ceph-mon[28361]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:06.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T01:26:06.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:06 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T01:26:06.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:06 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:07.675 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:07 smithi090 ceph-mon[35142]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:07.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:07 smithi090 ceph-mon[35142]: Deploying daemon osd.4 on smithi090 2024-09-21T01:26:07.676 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:07 smithi045 ceph-mon[28361]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:07 smithi045 ceph-mon[28361]: Deploying daemon osd.4 on smithi090 2024-09-21T01:26:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:09.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:09 smithi090 ceph-mon[35142]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:09.800 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:09 smithi045 ceph-mon[28361]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:11.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:11 smithi090 ceph-mon[35142]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:11 smithi045 ceph-mon[28361]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:12.817 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:12 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:12.817 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:12 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:12.817 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:12 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:12.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:12.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:12.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:13.697 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:13 smithi090 ceph-mon[35142]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:13.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:13 smithi045 ceph-mon[28361]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:15 smithi045 ceph-mon[28361]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:15 smithi090 ceph-mon[35142]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:16.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.363+0000 7f3c977fe640 1 -- 172.21.15.90:0/134888843 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ca0108570 con 0x7f3c8c07af30 2024-09-21T01:26:16.364 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 4 on host 'smithi090' 2024-09-21T01:26:16.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3c8c07af30 msgr2=0x7f3c8c07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:16.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3c8c07af30 0x7f3c8c07d440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3c9c02f870 tx=0x7f3c9c03c040 comp rx=0 tx=0).stop 2024-09-21T01:26:16.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 msgr2=0x7f3ca0199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:16.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 0x7f3ca0199a50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c9000c810 tx=0x7f3c9000ccd0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 shutdown_connections 2024-09-21T01:26:16.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3c8c07af30 0x7f3c8c07d440 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3ca0106630 0x7f3ca0199a50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 --2- 172.21.15.90:0/134888843 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3ca006c2f0 0x7f3ca01993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:16.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 >> 172.21.15.90:0/134888843 conn(0x7f3ca00fdca0 msgr2=0x7f3ca0100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:16.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 shutdown_connections 2024-09-21T01:26:16.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:16.365+0000 7f3ca84b6640 1 -- 172.21.15.90:0/134888843 wait complete. 2024-09-21T01:26:16.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:16.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:16.659 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:16.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:16.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:16.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:16.945 DEBUG:teuthology.orchestra.run.smithi090:osd.4> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.4.service 2024-09-21T01:26:16.948 INFO:tasks.cephadm:Deploying osd.5 on smithi090 with /dev/vg_nvme/lv_3... 2024-09-21T01:26:16.948 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T01:26:17.498 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:26:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:26:17.240+0000 7f7c78e91640 -1 osd.4 0 log_to_monitors true 2024-09-21T01:26:17.878 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:17 smithi090 ceph-mon[35142]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:17.878 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:17 smithi090 ceph-mon[35142]: from='osd.4 [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T01:26:17.878 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:17 smithi090 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T01:26:17.898 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:26:17.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:17 smithi045 ceph-mon[28361]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:17 smithi045 ceph-mon[28361]: from='osd.4 [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T01:26:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:17 smithi045 ceph-mon[28361]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T01:26:18.748 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:26:18.566+0000 7f7c74e13640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T01:26:18.749 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:26:18.570+0000 7f7c70c11640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:26:18.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T01:26:18.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-mon[35142]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T01:26:18.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:18.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-mon[35142]: from='osd.4 [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:18.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:18 smithi090 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:18.846 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:18 smithi045 ceph-mon[28361]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T01:26:18.846 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:18 smithi045 ceph-mon[28361]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T01:26:18.847 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:18 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:18.847 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:18 smithi045 ceph-mon[28361]: from='osd.4 [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:18.847 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:18 smithi045 ceph-mon[28361]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:19 smithi045 ceph-mon[28361]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:19 smithi045 ceph-mon[28361]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:26:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:19 smithi045 ceph-mon[28361]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T01:26:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:19 smithi090 ceph-mon[35142]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T01:26:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:19 smithi090 ceph-mon[35142]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:26:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:19 smithi090 ceph-mon[35142]: osdmap e31: 5 total, 4 up, 5 in 2024-09-21T01:26:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:20.741 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:26:20.741 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: osd.4 [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] boot 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.742 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:20 smithi090 ceph-mon[35142]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T01:26:20.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:26:20.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:26:20.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: osd.4 [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] boot 2024-09-21T01:26:20.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T01:26:20.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T01:26:20.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:20.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:20 smithi045 ceph-mon[28361]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T01:26:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:26:21.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-09-21T01:26:21.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:21 smithi090 ceph-mon[35142]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:21.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:21 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi090 2024-09-21T01:26:21.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:21 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi090 to 14912M 2024-09-21T01:26:21.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:21 smithi090 ceph-mon[35142]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T01:26:21.843 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:26:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:21 smithi045 ceph-mon[28361]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:21 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi090 2024-09-21T01:26:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:21 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi090 to 14912M 2024-09-21T01:26:21.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:21 smithi045 ceph-mon[28361]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T01:26:22.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.234+0000 7f3b88c5a640 1 -- 172.21.15.90:0/1355365056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 msgr2=0x7f3b84103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:22.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.234+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/1355365056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b84103880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009880 tx=0x7f3b7802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:26:22.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.234+0000 7f3b88c5a640 1 -- 172.21.15.90:0/1355365056 shutdown_connections 2024-09-21T01:26:22.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.234+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/1355365056 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b84103880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:22.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.235+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/1355365056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 0x7f3b841027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:22.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.235+0000 7f3b88c5a640 1 -- 172.21.15.90:0/1355365056 >> 172.21.15.90:0/1355365056 conn(0x7f3b840fdcb0 msgr2=0x7f3b84100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:22.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.235+0000 7f3b88c5a640 1 -- 172.21.15.90:0/1355365056 shutdown_connections 2024-09-21T01:26:22.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.235+0000 7f3b88c5a640 1 -- 172.21.15.90:0/1355365056 wait complete. 2024-09-21T01:26:22.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.235+0000 7f3b88c5a640 1 Processor -- start 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.235+0000 7f3b88c5a640 1 -- start start 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b88c5a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 0x7f3b84198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b88c5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b841989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b88c5a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8410b110 con 0x7f3b84102340 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b88c5a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8410af90 con 0x7f3b84103470 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b841989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b841989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:44970/0 (socket says 172.21.15.90:44970) 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 -- 172.21.15.90:0/2785581186 learned_addr learned my addr 172.21.15.90:0/2785581186 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b8259c640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 0x7f3b84198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:22.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 -- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 msgr2=0x7f3b84198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 0x7f3b84198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 -- 172.21.15.90:0/2785581186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b84199470 con 0x7f3b84103470 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b8259c640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 0x7f3b84198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b81d9b640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b841989a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b78000c00 tx=0x7f3b78004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3b7803f070 con 0x7f3b84103470 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3b78004440 con 0x7f3b84103470 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b84199700 con 0x7f3b84103470 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.236+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3b78035030 con 0x7f3b84103470 2024-09-21T01:26:22.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.237+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b84199c80 con 0x7f3b84103470 2024-09-21T01:26:22.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.238+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b841027b0 con 0x7f3b84103470 2024-09-21T01:26:22.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.239+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f3b78042070 con 0x7f3b84103470 2024-09-21T01:26:22.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.239+0000 7f3b737fe640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3b6c07af30 0x7f3b6c07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:22.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.239+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4206+0+0 (secure 0 0 0) 0x7f3b780be970 con 0x7f3b84103470 2024-09-21T01:26:22.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.239+0000 7f3b8259c640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3b6c07af30 0x7f3b6c07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:22.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.240+0000 7f3b8259c640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3b6c07af30 0x7f3b6c07d440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009b40 tx=0x7f3b74005e50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:22.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.243+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3b7808a700 con 0x7f3b84103470 2024-09-21T01:26:22.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:22.428+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3b84107f80 con 0x7f3b6c07af30 2024-09-21T01:26:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:26:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:26:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:22.923 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:22.923 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:26:22.923 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:26:22.923 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:23.825 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:23 smithi090 ceph-mon[35142]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:23.826 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:23 smithi090 ceph-mon[35142]: from='client.24225 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:23 smithi045 ceph-mon[28361]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:23 smithi045 ceph-mon[28361]: from='client.24225 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:24.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:24 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/502705333' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0122a5d6-f32b-4851-851e-aa054f411b75"} : dispatch 2024-09-21T01:26:24.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:24 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0122a5d6-f32b-4851-851e-aa054f411b75"} : dispatch 2024-09-21T01:26:24.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:24 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0122a5d6-f32b-4851-851e-aa054f411b75"}]': finished 2024-09-21T01:26:24.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:24 smithi090 ceph-mon[35142]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T01:26:24.895 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:24 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:24.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:24 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/502705333' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0122a5d6-f32b-4851-851e-aa054f411b75"} : dispatch 2024-09-21T01:26:24.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:24 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "0122a5d6-f32b-4851-851e-aa054f411b75"} : dispatch 2024-09-21T01:26:24.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:24 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0122a5d6-f32b-4851-851e-aa054f411b75"}]': finished 2024-09-21T01:26:24.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:24 smithi045 ceph-mon[28361]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T01:26:24.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:24 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:25.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:25 smithi045 ceph-mon[28361]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-21T01:26:25.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:25 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/1960559451' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:26:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:25 smithi090 ceph-mon[35142]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-21T01:26:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:25 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/1960559451' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:26:27.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:27 smithi045 ceph-mon[28361]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T01:26:27.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:27 smithi090 ceph-mon[35142]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T01:26:29.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:29 smithi090 ceph-mon[35142]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-21T01:26:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:29 smithi045 ceph-mon[28361]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-21T01:26:31.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:31 smithi090 ceph-mon[35142]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-21T01:26:31.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:31 smithi045 ceph-mon[28361]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-21T01:26:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:32 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T01:26:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:32 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:32.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:32 smithi090 ceph-mon[35142]: Deploying daemon osd.5 on smithi090 2024-09-21T01:26:32.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:32 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T01:26:32.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:32 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:32.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:32 smithi045 ceph-mon[28361]: Deploying daemon osd.5 on smithi090 2024-09-21T01:26:33.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:33 smithi045 ceph-mon[28361]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:26:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:33 smithi090 ceph-mon[35142]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:26:35.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:35 smithi045 ceph-mon[28361]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:35.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:35 smithi090 ceph-mon[35142]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:37 smithi045 ceph-mon[28361]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:37 smithi090 ceph-mon[35142]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:38.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:38.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:38.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:38.981 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:38.981 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:38.981 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:39.963 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:39 smithi090 ceph-mon[35142]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:39 smithi045 ceph-mon[28361]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:41.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.605+0000 7f3b737fe640 1 -- 172.21.15.90:0/2785581186 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3b84107f80 con 0x7f3b6c07af30 2024-09-21T01:26:41.609 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 5 on host 'smithi090' 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3b6c07af30 msgr2=0x7f3b6c07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3b6c07af30 0x7f3b6c07d440 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b74009b40 tx=0x7f3b74005e50 comp rx=0 tx=0).stop 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 msgr2=0x7f3b841989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b841989a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3b78000c00 tx=0x7f3b78004040 comp rx=0 tx=0).stop 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 shutdown_connections 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3b6c07af30 0x7f3b6c07d440 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3b84103470 0x7f3b841989a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 --2- 172.21.15.90:0/2785581186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3b84102340 0x7f3b84198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 >> 172.21.15.90:0/2785581186 conn(0x7f3b840fdcb0 msgr2=0x7f3b84106860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 shutdown_connections 2024-09-21T01:26:41.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:41.608+0000 7f3b88c5a640 1 -- 172.21.15.90:0/2785581186 wait complete. 2024-09-21T01:26:41.904 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:41 smithi090 ceph-mon[35142]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:41.904 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:41.904 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:41.904 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:41.904 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:41 smithi045 ceph-mon[28361]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:42.196 DEBUG:teuthology.orchestra.run.smithi090:osd.5> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.5.service 2024-09-21T01:26:42.198 INFO:tasks.cephadm:Deploying osd.6 on smithi090 with /dev/vg_nvme/lv_2... 2024-09-21T01:26:42.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T01:26:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:26:42.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:26:42 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:26:42.832+0000 7efde23e8640 -1 osd.5 0 log_to_monitors true 2024-09-21T01:26:43.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:43 smithi090 ceph-mon[35142]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:43.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:43 smithi090 ceph-mon[35142]: from='osd.5 [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T01:26:43.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:43 smithi090 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T01:26:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:43 smithi045 ceph-mon[28361]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:43 smithi045 ceph-mon[28361]: from='osd.5 [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T01:26:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:43 smithi045 ceph-mon[28361]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-21T01:26:44.877 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:26:44.808+0000 7efdde36a640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T01:26:44.877 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:26:44.814+0000 7efdda168640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:26:44.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T01:26:44.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-mon[35142]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T01:26:44.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:44.877 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-mon[35142]: from='osd.5 [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:44.878 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:44 smithi090 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:45.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:44 smithi045 ceph-mon[28361]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T01:26:45.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:44 smithi045 ceph-mon[28361]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T01:26:45.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:45.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:44 smithi045 ceph-mon[28361]: from='osd.5 [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:45.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:44 smithi045 ceph-mon[28361]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi090 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:26:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:26:46.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi090 to 7456M 2024-09-21T01:26:46.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:46.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:46.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T01:26:46.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi090 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi090 to 7456M 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:46.178 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:26:46.862 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-09-21T01:26:46.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: osd.5 [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] boot 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:46.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:46 smithi090 ceph-mon[35142]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T01:26:47.108 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:26:47.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:26:47.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:26:47.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: osd.5 [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] boot 2024-09-21T01:26:47.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T01:26:47.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T01:26:47.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:26:47.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:26:47.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:47.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:26:47.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:26:47.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:46 smithi045 ceph-mon[28361]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T01:26:47.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 -- 172.21.15.90:0/4170994724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 msgr2=0x7f23d0103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 --2- 172.21.15.90:0/4170994724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d0103890 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f23c4009880 tx=0x7f23c402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 -- 172.21.15.90:0/4170994724 shutdown_connections 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 --2- 172.21.15.90:0/4170994724 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d0103890 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 --2- 172.21.15.90:0/4170994724 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d0102350 0x7f23d01027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 -- 172.21.15.90:0/4170994724 >> 172.21.15.90:0/4170994724 conn(0x7f23d00fdce0 msgr2=0x7f23d0100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 -- 172.21.15.90:0/4170994724 shutdown_connections 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.529+0000 7f23d66d5640 1 -- 172.21.15.90:0/4170994724 wait complete. 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23d66d5640 1 Processor -- start 2024-09-21T01:26:47.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23d66d5640 1 -- start start 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23d66d5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d0102350 0x7f23d006b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23d66d5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d019e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23d66d5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f23d010b170 con 0x7f23d0102350 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23d66d5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f23d010aff0 con 0x7f23d0103480 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23cf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d019e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23cf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d019e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56272/0 (socket says 172.21.15.90:56272) 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.530+0000 7f23cf7fe640 1 -- 172.21.15.90:0/1649636298 learned_addr learned my addr 172.21.15.90:0/1649636298 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cffff640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d0102350 0x7f23d006b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cf7fe640 1 -- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d0102350 msgr2=0x7f23d006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cf7fe640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d0102350 0x7f23d006b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cf7fe640 1 -- 172.21.15.90:0/1649636298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d019e830 con 0x7f23d0103480 2024-09-21T01:26:47.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cf7fe640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d019e2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f23c4005940 tx=0x7f23c40048f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:47.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f23c403f070 con 0x7f23d0103480 2024-09-21T01:26:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f23c4003a80 con 0x7f23d0103480 2024-09-21T01:26:47.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d019ea00 con 0x7f23d0103480 2024-09-21T01:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f23c4035030 con 0x7f23d0103480 2024-09-21T01:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.531+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d019ef60 con 0x7f23d0103480 2024-09-21T01:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.532+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23d0102840 con 0x7f23d0103480 2024-09-21T01:26:47.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.533+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f23c4042070 con 0x7f23d0103480 2024-09-21T01:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.534+0000 7f23cd7fa640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f23b807af30 0x7f23b807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.534+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4638+0+0 (secure 0 0 0) 0x7f23c40be6b0 con 0x7f23d0103480 2024-09-21T01:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.534+0000 7f23cffff640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f23b807af30 0x7f23b807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:26:47.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.535+0000 7f23cffff640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f23b807af30 0x7f23b807d440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009990 tx=0x7f23c0009110 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:26:47.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.538+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f23c408a210 con 0x7f23d0103480 2024-09-21T01:26:47.725 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:26:47.724+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f23d0107fa0 con 0x7f23b807af30 2024-09-21T01:26:48.107 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:47 smithi090 ceph-mon[35142]: from='client.24243 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:48.107 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:47 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:26:48.107 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:47 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:26:48.107 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:47 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:47 smithi045 ceph-mon[28361]: from='client.24243 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:26:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:47 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:26:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:47 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:26:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:47 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:49.125 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:48 smithi090 ceph-mon[35142]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T01:26:49.125 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:48 smithi090 ceph-mon[35142]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:26:49.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:48 smithi045 ceph-mon[28361]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T01:26:49.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:48 smithi045 ceph-mon[28361]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:26:50.368 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:50 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/1848483828' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5170fdbb-6229-40f8-b415-de7d628a0490"} : dispatch 2024-09-21T01:26:50.368 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:50 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5170fdbb-6229-40f8-b415-de7d628a0490"} : dispatch 2024-09-21T01:26:50.368 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:50 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5170fdbb-6229-40f8-b415-de7d628a0490"}]': finished 2024-09-21T01:26:50.368 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:50 smithi090 ceph-mon[35142]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T01:26:50.368 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:50 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:26:50.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:50 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/1848483828' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5170fdbb-6229-40f8-b415-de7d628a0490"} : dispatch 2024-09-21T01:26:50.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:50 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "5170fdbb-6229-40f8-b415-de7d628a0490"} : dispatch 2024-09-21T01:26:50.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:50 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5170fdbb-6229-40f8-b415-de7d628a0490"}]': finished 2024-09-21T01:26:50.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:50 smithi045 ceph-mon[28361]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T01:26:50.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:50 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:26:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:51 smithi045 ceph-mon[28361]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:26:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:51 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/4238346565' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:26:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:51 smithi090 ceph-mon[35142]: pgmap v147: 1 pgs: 1 unknown; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:26:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:51 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/4238346565' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:26:53.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:53 smithi045 ceph-mon[28361]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T01:26:53.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:53.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:53 smithi090 ceph-mon[35142]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T01:26:53.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:26:55.403 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:55 smithi090 ceph-mon[35142]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T01:26:55.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:55 smithi045 ceph-mon[28361]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-21T01:26:57.243 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:57 smithi090 ceph-mon[35142]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T01:26:57.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:57 smithi045 ceph-mon[28361]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T01:26:58.191 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T01:26:58.191 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:58.191 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:58 smithi090 ceph-mon[35142]: Deploying daemon osd.6 on smithi090 2024-09-21T01:26:58.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T01:26:58.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:26:58.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:58 smithi045 ceph-mon[28361]: Deploying daemon osd.6 on smithi090 2024-09-21T01:26:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:26:59 smithi045 ceph-mon[28361]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:26:59.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:26:59 smithi090 ceph-mon[35142]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:27:01.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:01 smithi045 ceph-mon[28361]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:27:01.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:01 smithi090 ceph-mon[35142]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:27:03.410 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:03 smithi090 ceph-mon[35142]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T01:27:03.410 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:03 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:03.410 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:03 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:03.411 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:03 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:03.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:03 smithi045 ceph-mon[28361]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T01:27:03.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:03 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:03.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:03 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:03.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:03 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:05.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:05 smithi090 ceph-mon[35142]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:05.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:05 smithi045 ceph-mon[28361]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:06.282 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 6 on host 'smithi090' 2024-09-21T01:27:06.283 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.281+0000 7f23cd7fa640 1 -- 172.21.15.90:0/1649636298 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f23d0107fa0 con 0x7f23b807af30 2024-09-21T01:27:06.284 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f23b807af30 msgr2=0x7f23b807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f23b807af30 0x7f23b807d440 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f23c0009990 tx=0x7f23c0009110 comp rx=0 tx=0).stop 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 msgr2=0x7f23d019e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d019e2f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f23c4005940 tx=0x7f23c40048f0 comp rx=0 tx=0).stop 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 shutdown_connections 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f23b807af30 0x7f23b807d440 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23d0103480 0x7f23d019e2f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 --2- 172.21.15.90:0/1649636298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f23d0102350 0x7f23d006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 >> 172.21.15.90:0/1649636298 conn(0x7f23d00fdce0 msgr2=0x7f23d0106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 shutdown_connections 2024-09-21T01:27:06.285 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:06.284+0000 7f23d66d5640 1 -- 172.21.15.90:0/1649636298 wait complete. 2024-09-21T01:27:06.961 DEBUG:teuthology.orchestra.run.smithi090:osd.6> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.6.service 2024-09-21T01:27:06.964 INFO:tasks.cephadm:Deploying osd.7 on smithi090 with /dev/vg_nvme/lv_1... 2024-09-21T01:27:06.964 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T01:27:07.559 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:07 smithi090 ceph-mon[35142]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:07.559 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:07.559 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:07.572 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:27:07.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:07 smithi045 ceph-mon[28361]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:07.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:07.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:07.870 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:27:07 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:27:07.558+0000 7fb2806dc640 -1 osd.6 0 log_to_monitors true 2024-09-21T01:27:08.583 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:08.584 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:08 smithi090 ceph-mon[35142]: from='osd.6 [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T01:27:08.584 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:08 smithi090 ceph-mon[35142]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T01:27:08.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:08.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:08 smithi045 ceph-mon[28361]: from='osd.6 [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T01:27:08.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:08 smithi045 ceph-mon[28361]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T01:27:09.428 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-mon[35142]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:09.429 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-mon[35142]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T01:27:09.429 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-mon[35142]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T01:27:09.429 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:09.429 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-mon[35142]: from='osd.6 [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:09.429 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-mon[35142]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:09 smithi045 ceph-mon[28361]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:09 smithi045 ceph-mon[28361]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T01:27:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:09 smithi045 ceph-mon[28361]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T01:27:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:09 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:09 smithi045 ceph-mon[28361]: from='osd.6 [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:09.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:09 smithi045 ceph-mon[28361]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:10.021 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:27:09.768+0000 7fb27ce73640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T01:27:10.021 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:27:09 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:27:09.775+0000 7fb274454640 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:27:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:27:10.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:10.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:10 smithi045 ceph-mon[28361]: from='osd.6 ' entity='osd.6' 2024-09-21T01:27:10.715 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:27:10.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:10.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:27:10.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T01:27:10.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:10.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:10.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:10.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:10 smithi090 ceph-mon[35142]: from='osd.6 ' entity='osd.6' 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 -- 172.21.15.90:0/4189185185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850101820 msgr2=0x7fe850101c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 --2- 172.21.15.90:0/4189185185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850101820 0x7fe850101c70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe848009c20 tx=0x7fe84802a970 comp rx=0 tx=0).stop 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 -- 172.21.15.90:0/4189185185 shutdown_connections 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 --2- 172.21.15.90:0/4189185185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe850106900 0x7fe850108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 --2- 172.21.15.90:0/4189185185 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850101820 0x7fe850101c70 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 -- 172.21.15.90:0/4189185185 >> 172.21.15.90:0/4189185185 conn(0x7fe8500fd450 msgr2=0x7fe8500ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 -- 172.21.15.90:0/4189185185 shutdown_connections 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 -- 172.21.15.90:0/4189185185 wait complete. 2024-09-21T01:27:11.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.187+0000 7fe855be5640 1 Processor -- start 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe855be5640 1 -- start start 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe855be5640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe850101820 0x7fe850199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe855be5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 0x7fe850199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe855be5640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe85010c270 con 0x7fe850101820 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe855be5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe85010c0f0 con 0x7fe850106900 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 0x7fe850199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 0x7fe850199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55830/0 (socket says 172.21.15.90:55830) 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 -- 172.21.15.90:0/906528713 learned_addr learned my addr 172.21.15.90:0/906528713 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84f7fe640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe850101820 0x7fe850199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 -- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe850101820 msgr2=0x7fe850199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe850101820 0x7fe850199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 -- 172.21.15.90:0/906528713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85019a6b0 con 0x7fe850106900 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84effd640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 0x7fe850199be0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe83800b420 tx=0x7fe83800b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:11.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe838011630 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.188+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe838016070 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.189+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85019a9a0 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.189+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe838010b80 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.189+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85019af80 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.189+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe850101cf0 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.190+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fe8380117d0 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.191+0000 7fe84cff9640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe83c07af30 0x7fe83c07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.191+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5103+0+0 (secure 0 0 0) 0x7fe83809d430 con 0x7fe850106900 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.191+0000 7fe84f7fe640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe83c07af30 0x7fe83c07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.192+0000 7fe84f7fe640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe83c07af30 0x7fe83c07d440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe848009890 tx=0x7fe84803c040 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:11.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.195+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe838068dc0 con 0x7fe850106900 2024-09-21T01:27:11.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:11.387+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe8501088e0 con 0x7fe83c07af30 2024-09-21T01:27:11.388 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:11 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:27:11.388 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:11 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:27:11.388 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:11 smithi090 ceph-mon[35142]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:11.388 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:11 smithi090 ceph-mon[35142]: osd.6 [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] boot 2024-09-21T01:27:11.388 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:11 smithi090 ceph-mon[35142]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T01:27:11.388 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:11 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:11.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:11 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:27:11.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:11 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:27:11.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:11 smithi045 ceph-mon[28361]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T01:27:11.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:11 smithi045 ceph-mon[28361]: osd.6 [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] boot 2024-09-21T01:27:11.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:11 smithi045 ceph-mon[28361]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T01:27:11.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:11 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T01:27:12.592 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:12 smithi090 ceph-mon[35142]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T01:27:12.592 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:12 smithi090 ceph-mon[35142]: from='client.24261 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:12.592 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:12 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:27:12.592 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:12 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:27:12.592 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:12 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:12.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:12 smithi045 ceph-mon[28361]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T01:27:12.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:12 smithi045 ceph-mon[28361]: from='client.24261 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:27:12.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-21T01:27:12.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-21T01:27:12.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:12 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:13 smithi045 ceph-mon[28361]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-09-21T01:27:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:13 smithi045 ceph-mon[28361]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T01:27:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:13 smithi045 ceph-mon[28361]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T01:27:13.695 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:13 smithi090 ceph-mon[35142]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-09-21T01:27:13.695 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:13 smithi090 ceph-mon[35142]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T01:27:13.695 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:13 smithi090 ceph-mon[35142]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi090 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi090 to 4970M 2024-09-21T01:27:14.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:14.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi090 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi090 to 4970M 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:14.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:15 smithi045 ceph-mon[28361]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-09-21T01:27:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:15 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/1623150605' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9"} : dispatch 2024-09-21T01:27:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:15 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9"} : dispatch 2024-09-21T01:27:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:15 smithi045 ceph-mon[28361]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9"}]': finished 2024-09-21T01:27:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:15 smithi045 ceph-mon[28361]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T01:27:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:15 smithi090 ceph-mon[35142]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 513 MiB used, 625 GiB / 626 GiB avail 2024-09-21T01:27:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:15 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/1623150605' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9"} : dispatch 2024-09-21T01:27:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:15 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9"} : dispatch 2024-09-21T01:27:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:15 smithi090 ceph-mon[35142]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9"}]': finished 2024-09-21T01:27:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:15 smithi090 ceph-mon[35142]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T01:27:15.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:16.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:16 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/3117369991' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:27:16.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:16 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/3117369991' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-21T01:27:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:17 smithi045 ceph-mon[28361]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:17 smithi045 ceph-mon[28361]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T01:27:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:17 smithi045 ceph-mon[28361]: Cluster is now healthy 2024-09-21T01:27:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:17 smithi090 ceph-mon[35142]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:17.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:17 smithi090 ceph-mon[35142]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T01:27:17.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:17 smithi090 ceph-mon[35142]: Cluster is now healthy 2024-09-21T01:27:19.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:19 smithi090 ceph-mon[35142]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T01:27:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:19 smithi045 ceph-mon[28361]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-21T01:27:21.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:21 smithi045 ceph-mon[28361]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T01:27:21.795 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:21 smithi090 ceph-mon[35142]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T01:27:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:23.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:23 smithi090 ceph-mon[35142]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-21T01:27:23.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T01:27:23.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:23.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:23 smithi090 ceph-mon[35142]: Deploying daemon osd.7 on smithi090 2024-09-21T01:27:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:23 smithi045 ceph-mon[28361]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-21T01:27:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T01:27:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:23 smithi045 ceph-mon[28361]: Deploying daemon osd.7 on smithi090 2024-09-21T01:27:25.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:25 smithi090 ceph-mon[35142]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:27:25.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:25 smithi045 ceph-mon[28361]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T01:27:27.656 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:27 smithi090 ceph-mon[35142]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T01:27:27.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:27 smithi045 ceph-mon[28361]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-21T01:27:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:29 smithi090 ceph-mon[35142]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:29.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:29.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:29 smithi045 ceph-mon[28361]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:29.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:31.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:31 smithi090 ceph-mon[35142]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:31.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:31 smithi045 ceph-mon[28361]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:32.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.610+0000 7fe84cff9640 1 -- 172.21.15.90:0/906528713 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe8501088e0 con 0x7fe83c07af30 2024-09-21T01:27:32.611 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 7 on host 'smithi090' 2024-09-21T01:27:32.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe83c07af30 msgr2=0x7fe83c07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:32.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe83c07af30 0x7fe83c07d440 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe848009890 tx=0x7fe84803c040 comp rx=0 tx=0).stop 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 msgr2=0x7fe850199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 0x7fe850199be0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe83800b420 tx=0x7fe83800b8e0 comp rx=0 tx=0).stop 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 shutdown_connections 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe83c07af30 0x7fe83c07d440 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe850106900 0x7fe850199be0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 --2- 172.21.15.90:0/906528713 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe850101820 0x7fe850199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 >> 172.21.15.90:0/906528713 conn(0x7fe8500fd450 msgr2=0x7fe8500ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 shutdown_connections 2024-09-21T01:27:32.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:32.613+0000 7fe855be5640 1 -- 172.21.15.90:0/906528713 wait complete. 2024-09-21T01:27:33.199 DEBUG:teuthology.orchestra.run.smithi090:osd.7> sudo journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.7.service 2024-09-21T01:27:33.201 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T01:27:33.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T01:27:33.479 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:33.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:33 smithi090 ceph-mon[35142]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:33.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:33 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.855 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:27:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:27:33.682+0000 7fc0b47fe640 -1 osd.7 0 log_to_monitors true 2024-09-21T01:27:33.877 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:33 smithi045 ceph-mon[28361]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:33.877 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.877 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.877 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.877 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:33 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:33.914 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 -- 172.21.15.45:0/2073854094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc103490 msgr2=0x7f53cc1038a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 --2- 172.21.15.45:0/2073854094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc103490 0x7f53cc1038a0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f53c0009880 tx=0x7f53c002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 -- 172.21.15.45:0/2073854094 shutdown_connections 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 --2- 172.21.15.45:0/2073854094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc103490 0x7f53cc1038a0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 --2- 172.21.15.45:0/2073854094 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53cc102360 0x7f53cc1027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 -- 172.21.15.45:0/2073854094 >> 172.21.15.45:0/2073854094 conn(0x7f53cc0fdcd0 msgr2=0x7f53cc100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 -- 172.21.15.45:0/2073854094 shutdown_connections 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.912+0000 7f53d13ca640 1 -- 172.21.15.45:0/2073854094 wait complete. 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53d13ca640 1 Processor -- start 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53d13ca640 1 -- start start 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53d13ca640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 0x7f53cc105f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53d13ca640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53cc103490 0x7f53cc1048c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53d13ca640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10b180 con 0x7f53cc102360 2024-09-21T01:27:33.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53d13ca640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f53cc10b000 con 0x7f53cc103490 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53caffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 0x7f53cc105f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53caffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 0x7f53cc105f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55972/0 (socket says 172.21.15.45:55972) 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53caffd640 1 -- 172.21.15.45:0/1273718658 learned_addr learned my addr 172.21.15.45:0/1273718658 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.913+0000 7f53ca7fc640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53cc103490 0x7f53cc1048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53caffd640 1 -- 172.21.15.45:0/1273718658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53cc103490 msgr2=0x7f53cc1048c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53caffd640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53cc103490 0x7f53cc1048c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53caffd640 1 -- 172.21.15.45:0/1273718658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53cc106790 con 0x7f53cc102360 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53caffd640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 0x7f53cc105f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00b420 tx=0x7f53bc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f53bc011630 con 0x7f53cc102360 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f53bc016070 con 0x7f53cc102360 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f53bc010b80 con 0x7f53cc102360 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53cc104f20 con 0x7f53cc102360 2024-09-21T01:27:33.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.914+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53cc1054e0 con 0x7f53cc102360 2024-09-21T01:27:33.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.915+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53cc10b8d0 con 0x7f53cc102360 2024-09-21T01:27:33.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.917+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f53bc01b460 con 0x7f53cc102360 2024-09-21T01:27:33.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.917+0000 7f53b3fff640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f53b407af80 0x7f53b407d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:33.917 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.917+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5185+0+0 (secure 0 0 0) 0x7f53bc09d420 con 0x7f53cc102360 2024-09-21T01:27:33.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.917+0000 7f53ca7fc640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f53b407af80 0x7f53b407d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:33.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.918+0000 7f53ca7fc640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f53b407af80 0x7f53b407d490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f53c0004d10 tx=0x7f53c0004d70 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:33.922 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:33.922+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f53bc068dd0 con 0x7f53cc102360 2024-09-21T01:27:34.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.111+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f53cc1a1b20 con 0x7f53cc102360 2024-09-21T01:27:34.111 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.111+0000 7f53b3fff640 1 -- 172.21.15.45:0/1273718658 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v47) ==== 74+0+130 (secure 0 0 0) 0x7f53bc06dc80 con 0x7f53cc102360 2024-09-21T01:27:34.111 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f53b407af80 msgr2=0x7f53b407d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f53b407af80 0x7f53b407d490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f53c0004d10 tx=0x7f53c0004d70 comp rx=0 tx=0).stop 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 msgr2=0x7f53cc105f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 0x7f53cc105f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f53bc00b420 tx=0x7f53bc00b8e0 comp rx=0 tx=0).stop 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 shutdown_connections 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f53b407af80 0x7f53b407d490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f53cc103490 0x7f53cc1048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 --2- 172.21.15.45:0/1273718658 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f53cc102360 0x7f53cc105f80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.113+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 >> 172.21.15.45:0/1273718658 conn(0x7f53cc0fdcd0 msgr2=0x7f53cc0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.114+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 shutdown_connections 2024-09-21T01:27:34.114 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:34.114+0000 7f53d13ca640 1 -- 172.21.15.45:0/1273718658 wait complete. 2024-09-21T01:27:34.856 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:34 smithi045 ceph-mon[28361]: from='osd.7 [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T01:27:34.856 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:34 smithi045 ceph-mon[28361]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T01:27:34.856 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:34 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1273718658' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T01:27:34.884 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":47,"num_osds":8,"num_up_osds":7,"osd_up_since":1726882030,"num_in_osds":8,"osd_in_since":1726882034,"num_remapped_pgs":0} 2024-09-21T01:27:34.908 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:34 smithi090 ceph-mon[35142]: from='osd.7 [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T01:27:34.908 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:34 smithi090 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-21T01:27:34.908 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:34 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1273718658' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T01:27:35.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:27:35.621+0000 7fc0b0780640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T01:27:35.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:27:35.627+0000 7fc0ac57e640 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T01:27:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-mon[35142]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T01:27:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-mon[35142]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T01:27:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-mon[35142]: from='osd.7 [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:35.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:35 smithi090 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:35.885 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-21T01:27:35.915 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:35 smithi045 ceph-mon[28361]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:35.915 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:35 smithi045 ceph-mon[28361]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T01:27:35.915 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:35 smithi045 ceph-mon[28361]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T01:27:35.915 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:35.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:35 smithi045 ceph-mon[28361]: from='osd.7 [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:35.916 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:35 smithi045 ceph-mon[28361]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]} : dispatch 2024-09-21T01:27:36.162 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:36.593 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 -- 172.21.15.45:0/43367503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748102350 msgr2=0x7fc7481027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 --2- 172.21.15.45:0/43367503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748102350 0x7fc7481027c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc738009880 tx=0x7fc73802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 -- 172.21.15.45:0/43367503 shutdown_connections 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 --2- 172.21.15.45:0/43367503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc748103480 0x7fc748103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 --2- 172.21.15.45:0/43367503 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748102350 0x7fc7481027c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 -- 172.21.15.45:0/43367503 >> 172.21.15.45:0/43367503 conn(0x7fc7480fdce0 msgr2=0x7fc748100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 -- 172.21.15.45:0/43367503 shutdown_connections 2024-09-21T01:27:36.594 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.593+0000 7fc74c837640 1 -- 172.21.15.45:0/43367503 wait complete. 2024-09-21T01:27:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74c837640 1 Processor -- start 2024-09-21T01:27:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74c837640 1 -- start start 2024-09-21T01:27:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74c837640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc748102350 0x7fc748199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74c837640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 0x7fc748199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74c837640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc74810b110 con 0x7fc748103480 2024-09-21T01:27:36.596 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74c837640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc74810af90 con 0x7fc748102350 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc745d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 0x7fc748199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc745d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 0x7fc748199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55776/0 (socket says 172.21.15.45:55776) 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc745d9b640 1 -- 172.21.15.45:0/502525614 learned_addr learned my addr 172.21.15.45:0/502525614 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.594+0000 7fc74659c640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc748102350 0x7fc748199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc745d9b640 1 -- 172.21.15.45:0/502525614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc748102350 msgr2=0x7fc748199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc745d9b640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc748102350 0x7fc748199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc745d9b640 1 -- 172.21.15.45:0/502525614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74819a5c0 con 0x7fc748103480 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc745d9b640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 0x7fc748199af0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e970 tx=0x7fc73c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc73c00cac0 con 0x7fc748103480 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74819a8b0 con 0x7fc748103480 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc73c0054b0 con 0x7fc748103480 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc73c004c40 con 0x7fc748103480 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.595+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc74819ae90 con 0x7fc748103480 2024-09-21T01:27:36.597 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.596+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7481027c0 con 0x7fc748103480 2024-09-21T01:27:36.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.597+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fc73c00cc60 con 0x7fc748103480 2024-09-21T01:27:36.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.598+0000 7fc7377fe640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc73007af30 0x7fc73007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:36.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.598+0000 7fc74659c640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc73007af30 0x7fc73007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:36.598 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.598+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5222+0+0 (secure 0 0 0) 0x7fc73c09e370 con 0x7fc748103480 2024-09-21T01:27:36.599 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.598+0000 7fc74659c640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc73007af30 0x7fc73007d440 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc738004dd0 tx=0x7fc738005d50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:36.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.603+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc73c069c80 con 0x7fc748103480 2024-09-21T01:27:36.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.832+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fc74819b1a0 con 0x7fc748103480 2024-09-21T01:27:36.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: osdmap e49: 8 total, 7 up, 8 in 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:36.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: osd.7 [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] boot 2024-09-21T01:27:36.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T01:27:36.834 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:36.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.832+0000 7fc7377fe640 1 -- 172.21.15.45:0/502525614 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v50) ==== 74+0+130 (secure 0 0 0) 0x7fc73c06eb30 con 0x7fc748103480 2024-09-21T01:27:36.834 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:36.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc73007af30 msgr2=0x7fc73007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:36.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc73007af30 0x7fc73007d440 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc738004dd0 tx=0x7fc738005d50 comp rx=0 tx=0).stop 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 msgr2=0x7fc748199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 0x7fc748199af0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00e970 tx=0x7fc73c00ed70 comp rx=0 tx=0).stop 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 shutdown_connections 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc73007af30 0x7fc73007d440 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc748103480 0x7fc748199af0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 --2- 172.21.15.45:0/502525614 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc748102350 0x7fc748199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 >> 172.21.15.45:0/502525614 conn(0x7fc7480fdce0 msgr2=0x7fc7480ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.835+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 shutdown_connections 2024-09-21T01:27:36.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:36.836+0000 7fc74c837640 1 -- 172.21.15.45:0/502525614 wait complete. 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: osdmap e49: 8 total, 7 up, 8 in 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:36.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: osd.7 [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] boot 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T01:27:37.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T01:27:37.408 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":50,"num_osds":8,"num_up_osds":8,"osd_up_since":1726882056,"num_in_osds":8,"osd_in_since":1726882034,"num_remapped_pgs":1} 2024-09-21T01:27:37.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:27:37.718 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: purged_snaps scrub starts 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: purged_snaps scrub ok 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: Detected new or changed devices on smithi090 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi090 to 3728M 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:37.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:37.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/502525614' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T01:27:37.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:37.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:37 smithi045 ceph-mon[28361]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T01:27:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: purged_snaps scrub starts 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: purged_snaps scrub ok 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: Detected new or changed devices on smithi090 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi090 to 3728M 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/502525614' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:37 smithi090 ceph-mon[35142]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 -- 172.21.15.45:0/909220180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 msgr2=0x7f3d8c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/909220180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c102310 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3d88009880 tx=0x7f3d8802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 -- 172.21.15.45:0/909220180 shutdown_connections 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/909220180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 0x7f3d8c107c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/909220180 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c102310 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 -- 172.21.15.45:0/909220180 >> 172.21.15.45:0/909220180 conn(0x7f3d8c079bd0 msgr2=0x7f3d8c079fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 -- 172.21.15.45:0/909220180 shutdown_connections 2024-09-21T01:27:38.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.133+0000 7f3d93ac8640 1 -- 172.21.15.45:0/909220180 wait complete. 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d93ac8640 1 Processor -- start 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d93ac8640 1 -- start start 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d93ac8640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d93ac8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 0x7f3d8c199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d93ac8640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8c10efd0 con 0x7f3d8c101f00 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d93ac8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3d8c10ee50 con 0x7f3d8c107840 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d91864640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d91864640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55798/0 (socket says 172.21.15.45:55798) 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.134+0000 7f3d91864640 1 -- 172.21.15.45:0/3267383859 learned_addr learned my addr 172.21.15.45:0/3267383859 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:38.136 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d91063640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 0x7f3d8c199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d91864640 1 -- 172.21.15.45:0/3267383859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 msgr2=0x7f3d8c199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d91864640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 0x7f3d8c199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d91864640 1 -- 172.21.15.45:0/3267383859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d8c19f1b0 con 0x7f3d8c101f00 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d91063640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 0x7f3d8c199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d91864640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c199970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3d88006010 tx=0x7f3d880045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3d8803f070 con 0x7f3d8c101f00 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3d880049d0 con 0x7f3d8c101f00 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3d88035030 con 0x7f3d8c101f00 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d8c19f440 con 0x7f3d8c101f00 2024-09-21T01:27:38.137 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.135+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d8c19f970 con 0x7f3d8c101f00 2024-09-21T01:27:38.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.136+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d8c103180 con 0x7f3d8c101f00 2024-09-21T01:27:38.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.137+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f3d88042070 con 0x7f3d8c101f00 2024-09-21T01:27:38.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.138+0000 7f3d82ffd640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3d7807af30 0x7f3d7807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:38.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.138+0000 7f3d91063640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3d7807af30 0x7f3d7807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:38.138 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.138+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5502+0+0 (secure 0 0 0) 0x7f3d880bea00 con 0x7f3d8c101f00 2024-09-21T01:27:38.139 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.138+0000 7f3d91063640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3d7807af30 0x7f3d7807d440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c005fd0 tx=0x7f3d7c005ea0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:38.143 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.143+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3d8808a200 con 0x7f3d8c101f00 2024-09-21T01:27:38.330 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.330+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d8c10b700 con 0x7f3d8c101f00 2024-09-21T01:27:38.331 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.331+0000 7f3d82ffd640 1 -- 172.21.15.45:0/3267383859 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+13870 (secure 0 0 0) 0x7f3d8807fe20 con 0x7f3d8c101f00 2024-09-21T01:27:38.331 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:38.331 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":51,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","created":"2024-09-21T01:20:38.644738+0000","modified":"2024-09-21T01:27:37.611664+0000","last_up_change":"2024-09-21T01:27:36.610150+0000","last_in_change":"2024-09-21T01:27:14.953689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:25:30.425195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20b68c0-eefc-40c0-b952-799c83b7f4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6803","nonce":3771776284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6805","nonce":3771776284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6809","nonce":3771776284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6807","nonce":3771776284}]},"public_addr":"172.21.15.45:6803/3771776284","cluster_addr":"172.21.15.45:6805/3771776284","heartbeat_back_addr":"172.21.15.45:6809/3771776284","heartbeat_front_addr":"172.21.15.45:6807/3771776284","state":["exists","up"]},{"osd":1,"uuid":"b10dd316-1566-4bff-933b-1b96b16a3378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6811","nonce":2203931892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6813","nonce":2203931892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6817","nonce":2203931892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6815","nonce":2203931892}]},"public_addr":"172.21.15.45:6811/2203931892","cluster_addr":"172.21.15.45:6813/2203931892","heartbeat_back_addr":"172.21.15.45:6817/2203931892","heartbeat_front_addr":"172.21.15.45:6815/2203931892","state":["exists","up"]},{"osd":2,"uuid":"80cb32e4-abb7-4cd6-b2af-6e254e3c592b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6819","nonce":2681427309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6821","nonce":2681427309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6825","nonce":2681427309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6823","nonce":2681427309}]},"public_addr":"172.21.15.45:6819/2681427309","cluster_addr":"172.21.15.45:6821/2681427309","heartbeat_back_addr":"172.21.15.45:6825/2681427309","heartbeat_front_addr":"172.21.15.45:6823/2681427309","state":["exists","up"]},{"osd":3,"uuid":"1aaec02f-d282-4505-a860-3d77532911df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6827","nonce":2480895441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6829","nonce":2480895441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6833","nonce":2480895441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6831","nonce":2480895441}]},"public_addr":"172.21.15.45:6827/2480895441","cluster_addr":"172.21.15.45:6829/2480895441","heartbeat_back_addr":"172.21.15.45:6833/2480895441","heartbeat_front_addr":"172.21.15.45:6831/2480895441","state":["exists","up"]},{"osd":4,"uuid":"85e89409-05b7-432d-9044-3b730f4f68b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6801","nonce":2331499915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6803","nonce":2331499915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6807","nonce":2331499915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6805","nonce":2331499915}]},"public_addr":"172.21.15.90:6801/2331499915","cluster_addr":"172.21.15.90:6803/2331499915","heartbeat_back_addr":"172.21.15.90:6807/2331499915","heartbeat_front_addr":"172.21.15.90:6805/2331499915","state":["exists","up"]},{"osd":5,"uuid":"0122a5d6-f32b-4851-851e-aa054f411b75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6809","nonce":1613366233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6811","nonce":1613366233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6815","nonce":1613366233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6813","nonce":1613366233}]},"public_addr":"172.21.15.90:6809/1613366233","cluster_addr":"172.21.15.90:6811/1613366233","heartbeat_back_addr":"172.21.15.90:6815/1613366233","heartbeat_front_addr":"172.21.15.90:6813/1613366233","state":["exists","up"]},{"osd":6,"uuid":"5170fdbb-6229-40f8-b415-de7d628a0490","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6817","nonce":2608695333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6819","nonce":2608695333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6823","nonce":2608695333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6821","nonce":2608695333}]},"public_addr":"172.21.15.90:6817/2608695333","cluster_addr":"172.21.15.90:6819/2608695333","heartbeat_back_addr":"172.21.15.90:6823/2608695333","heartbeat_front_addr":"172.21.15.90:6821/2608695333","state":["exists","up"]},{"osd":7,"uuid":"3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6825","nonce":2703954098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6827","nonce":2703954098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6831","nonce":2703954098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6829","nonce":2703954098}]},"public_addr":"172.21.15.90:6825/2703954098","cluster_addr":"172.21.15.90:6827/2703954098","heartbeat_back_addr":"172.21.15.90:6831/2703954098","heartbeat_front_addr":"172.21.15.90:6829/2703954098","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:24:32.659134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:00.007423+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:27.198145+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:54.790013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:18.251812+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:43.822784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:08.545224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:34.701933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/1147989706":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1797066051":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3851449798":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/3707884113":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3194963614":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6800/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1334988077":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6800/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/1726148947":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/308996383":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6801/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:6800/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:0/1437053407":"2024-09-22T01:21:45.172023+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:27:38.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3d7807af30 msgr2=0x7f3d7807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:38.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3d7807af30 0x7f3d7807d440 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d7c005fd0 tx=0x7f3d7c005ea0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.333 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 msgr2=0x7f3d8c199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c199970 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3d88006010 tx=0x7f3d880045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 shutdown_connections 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3d7807af30 0x7f3d7807d440 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3d8c107840 0x7f3d8c199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 --2- 172.21.15.45:0/3267383859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3d8c101f00 0x7f3d8c199970 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 >> 172.21.15.45:0/3267383859 conn(0x7f3d8c079bd0 msgr2=0x7f3d8c1088d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 shutdown_connections 2024-09-21T01:27:38.334 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:38.333+0000 7f3d93ac8640 1 -- 172.21.15.45:0/3267383859 wait complete. 2024-09-21T01:27:38.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:38 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3267383859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:27:38.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:38 smithi045 ceph-mon[28361]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T01:27:38.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:38 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3267383859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:27:38.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:38 smithi090 ceph-mon[35142]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T01:27:39.037 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T01:25:30.425195+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T01:27:39.037 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T01:27:39.300 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:39 smithi045 ceph-mon[28361]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:39.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 -- 172.21.15.45:0/1711734942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230102750 msgr2=0x7f3230102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:39.723 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 --2- 172.21.15.45:0/1711734942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230102750 0x7f3230102b60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f322c009880 tx=0x7f322c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 -- 172.21.15.45:0/1711734942 shutdown_connections 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 --2- 172.21.15.45:0/1711734942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230108090 0x7f32301084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 --2- 172.21.15.45:0/1711734942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230102750 0x7f3230102b60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 -- 172.21.15.45:0/1711734942 >> 172.21.15.45:0/1711734942 conn(0x7f32300fe1a0 msgr2=0x7f32301005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 -- 172.21.15.45:0/1711734942 shutdown_connections 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.719+0000 7f32387c1640 1 -- 172.21.15.45:0/1711734942 wait complete. 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f32387c1640 1 Processor -- start 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f32387c1640 1 -- start start 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f32387c1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230102750 0x7f32300fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f32387c1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 0x7f32300fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f32387c1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f323010f820 con 0x7f3230108090 2024-09-21T01:27:39.724 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f32387c1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f323010f6a0 con 0x7f3230102750 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f3235d5c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 0x7f32300fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f3235d5c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 0x7f32300fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55818/0 (socket says 172.21.15.45:55818) 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.720+0000 7f3235d5c640 1 -- 172.21.15.45:0/1683481172 learned_addr learned my addr 172.21.15.45:0/1683481172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f323655d640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230102750 0x7f32300fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f3235d5c640 1 -- 172.21.15.45:0/1683481172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230102750 msgr2=0x7f32300fe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f3235d5c640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230102750 0x7f32300fe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f3235d5c640 1 -- 172.21.15.45:0/1683481172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3230100560 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f323655d640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230102750 0x7f32300fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f3235d5c640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 0x7f32300fec10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f322000d6d0 tx=0x7f322000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3220017070 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f32200054b0 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f32200049d0 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32300ff210 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.721+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32301a5cc0 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.722+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32301039d0 con 0x7f3230108090 2024-09-21T01:27:39.725 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.723+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f3220004e90 con 0x7f3230108090 2024-09-21T01:27:39.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.724+0000 7f32277fe640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f321c07af80 0x7f321c07d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:39.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.724+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f322009dae0 con 0x7f3230108090 2024-09-21T01:27:39.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.724+0000 7f323655d640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f321c07af80 0x7f321c07d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:39.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.725+0000 7f323655d640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f321c07af80 0x7f321c07d490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f322c000c00 tx=0x7f322c005a20 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:39.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.729+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3220069360 con 0x7f3230108090 2024-09-21T01:27:39.915 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.915+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f32301a5e90 con 0x7f3230108090 2024-09-21T01:27:39.916 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.915+0000 7f32277fe640 1 -- 172.21.15.45:0/1683481172 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7f322006e210 con 0x7f3230108090 2024-09-21T01:27:39.916 INFO:teuthology.orchestra.run.smithi045.stdout:pg_num: 1 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.917+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f321c07af80 msgr2=0x7f321c07d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.917+0000 7f32387c1640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f321c07af80 0x7f321c07d490 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f322c000c00 tx=0x7f322c005a20 comp rx=0 tx=0).stop 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.917+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 msgr2=0x7f32300fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.917+0000 7f32387c1640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 0x7f32300fec10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f322000d6d0 tx=0x7f322000db90 comp rx=0 tx=0).stop 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 shutdown_connections 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f321c07af80 0x7f321c07d490 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3230108090 0x7f32300fec10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.918 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 --2- 172.21.15.45:0/1683481172 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3230102750 0x7f32300fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 >> 172.21.15.45:0/1683481172 conn(0x7f32300fe1a0 msgr2=0x7f3230101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 shutdown_connections 2024-09-21T01:27:39.919 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:39.918+0000 7f32387c1640 1 -- 172.21.15.45:0/1683481172 wait complete. 2024-09-21T01:27:39.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:39 smithi090 ceph-mon[35142]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:40.525 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T01:27:40.525 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T01:27:40.786 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:40.907 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:40 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1683481172' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T01:27:40.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:40 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1683481172' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 -- 172.21.15.45:0/1091994323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc103460 msgr2=0x7fb5cc1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 --2- 172.21.15.45:0/1091994323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc103460 0x7fb5cc1038d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009880 tx=0x7fb5c802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 -- 172.21.15.45:0/1091994323 shutdown_connections 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 --2- 172.21.15.45:0/1091994323 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5cc104590 0x7fb5cc1049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 --2- 172.21.15.45:0/1091994323 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc103460 0x7fb5cc1038d0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 -- 172.21.15.45:0/1091994323 >> 172.21.15.45:0/1091994323 conn(0x7fb5cc0fedd0 msgr2=0x7fb5cc101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 -- 172.21.15.45:0/1091994323 shutdown_connections 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.248+0000 7fb5d464f640 1 -- 172.21.15.45:0/1091994323 wait complete. 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d464f640 1 Processor -- start 2024-09-21T01:27:41.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d464f640 1 -- start start 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d464f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5cc103460 0x7fb5cc199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d464f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 0x7fb5cc199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d464f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c200 con 0x7fb5cc104590 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d464f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb5cc10c080 con 0x7fb5cc103460 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d1bea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 0x7fb5cc199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d1bea640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 0x7fb5cc199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55842/0 (socket says 172.21.15.45:55842) 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d1bea640 1 -- 172.21.15.45:0/3719291897 learned_addr learned my addr 172.21.15.45:0/3719291897 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d23eb640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5cc103460 0x7fb5cc199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d1bea640 1 -- 172.21.15.45:0/3719291897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5cc103460 msgr2=0x7fb5cc199410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d1bea640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5cc103460 0x7fb5cc199410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.249+0000 7fb5d1bea640 1 -- 172.21.15.45:0/3719291897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5cc19a550 con 0x7fb5cc104590 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.250+0000 7fb5d1bea640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 0x7fb5cc199a80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00d8d0 tx=0x7fb5bc00dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:41.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.250+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb5bc017070 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.250+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb5bc010460 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.250+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5cc19a840 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.250+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb5bc00f5f0 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.250+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5cc19ae20 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.251+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5cc1038d0 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.252+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fb5bc010610 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.252+0000 7fb5c37fe640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb5b807af30 0x7fb5b807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.252+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fb5bc09dbc0 con 0x7fb5cc104590 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.252+0000 7fb5d23eb640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb5b807af30 0x7fb5b807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.253+0000 7fb5d23eb640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb5b807af30 0x7fb5b807d440 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8000c00 tx=0x7fb5c8004cd0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:41.274 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.257+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb5bc0693c0 con 0x7fb5cc104590 2024-09-21T01:27:41.517 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.517+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fb5cc19f720 con 0x7fb5cc104590 2024-09-21T01:27:41.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.520+0000 7fb5c37fe640 1 -- 172.21.15.45:0/3719291897 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) ==== 170+0+59 (secure 0 0 0) 0x7fb5bc06e270 con 0x7fb5cc104590 2024-09-21T01:27:41.520 INFO:teuthology.orchestra.run.smithi045.stdout:[client.0] 2024-09-21T01:27:41.521 INFO:teuthology.orchestra.run.smithi045.stdout: key = AQANIe5mFknsHhAA6YAd+yy13xN04OU5jaJ5hQ== 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.522+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb5b807af30 msgr2=0x7fb5b807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.522+0000 7fb5d464f640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb5b807af30 0x7fb5b807d440 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8000c00 tx=0x7fb5c8004cd0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.522+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 msgr2=0x7fb5cc199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 0x7fb5cc199a80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc00d8d0 tx=0x7fb5bc00dd90 comp rx=0 tx=0).stop 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 shutdown_connections 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb5b807af30 0x7fb5b807d440 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb5cc104590 0x7fb5cc199a80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 --2- 172.21.15.45:0/3719291897 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5cc103460 0x7fb5cc199410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:41.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 >> 172.21.15.45:0/3719291897 conn(0x7fb5cc0fedd0 msgr2=0x7fb5cc1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:41.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 shutdown_connections 2024-09-21T01:27:41.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:41.523+0000 7fb5d464f640 1 -- 172.21.15.45:0/3719291897 wait complete. 2024-09-21T01:27:41.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:41 smithi045 ceph-mon[28361]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:41.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3719291897' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:27:41.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3719291897' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:27:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:41 smithi090 ceph-mon[35142]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:41.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:41 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3719291897' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:27:41.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:41 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3719291897' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:27:42.146 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:27:42.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T01:27:42.146 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T01:27:42.184 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T01:27:42.444 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi090/config 2024-09-21T01:27:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 -- 172.21.15.90:0/814888120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188103450 msgr2=0x7fa1881038c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 --2- 172.21.15.90:0/814888120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188103450 0x7fa1881038c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa184009880 tx=0x7fa18402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 -- 172.21.15.90:0/814888120 shutdown_connections 2024-09-21T01:27:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 --2- 172.21.15.90:0/814888120 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188104580 0x7fa188104990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 --2- 172.21.15.90:0/814888120 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188103450 0x7fa1881038c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.946 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 -- 172.21.15.90:0/814888120 >> 172.21.15.90:0/814888120 conn(0x7fa1880fedc0 msgr2=0x7fa188101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 -- 172.21.15.90:0/814888120 shutdown_connections 2024-09-21T01:27:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 -- 172.21.15.90:0/814888120 wait complete. 2024-09-21T01:27:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 Processor -- start 2024-09-21T01:27:42.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.945+0000 7fa1907de640 1 -- start start 2024-09-21T01:27:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa1907de640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188103450 0x7fa1881994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa1907de640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 0x7fa188199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa1907de640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa18810c270 con 0x7fa188103450 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa1907de640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa18810c0f0 con 0x7fa188104580 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 0x7fa188199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 0x7fa188199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48714/0 (socket says 172.21.15.90:48714) 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 -- 172.21.15.90:0/1953032616 learned_addr learned my addr 172.21.15.90:0/1953032616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18e57a640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188103450 0x7fa1881994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 -- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188103450 msgr2=0x7fa1881994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188103450 0x7fa1881994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 -- 172.21.15.90:0/1953032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa18819a630 con 0x7fa188104580 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18e57a640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188103450 0x7fa1881994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa18dd79640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 0x7fa188199b60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa17800b420 tx=0x7fa17800b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa178011630 con 0x7fa188104580 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa178016070 con 0x7fa188104580 2024-09-21T01:27:42.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa178010b80 con 0x7fa188104580 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.946+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa18819a920 con 0x7fa188104580 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.947+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa18819af00 con 0x7fa188104580 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.948+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa18810c9c0 con 0x7fa188104580 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.948+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fa1780117d0 con 0x7fa188104580 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.949+0000 7fa17f7fe640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa17407af40 0x7fa17407d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.949+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fa17809d670 con 0x7fa188104580 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.949+0000 7fa18e57a640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa17407af40 0x7fa17407d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:42.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.950+0000 7fa18e57a640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa17407af40 0x7fa17407d450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa184004dd0 tx=0x7fa184004cd0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:42.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:42.954+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa178068e70 con 0x7fa188104580 2024-09-21T01:27:43.198 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.197+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fa188103940 con 0x7fa188104580 2024-09-21T01:27:43.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.203+0000 7fa17f7fe640 1 -- 172.21.15.90:0/1953032616 <== mon.1 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v19) ==== 170+0+59 (secure 0 0 0) 0x7fa17806dd20 con 0x7fa188104580 2024-09-21T01:27:43.204 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2024-09-21T01:27:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQAPIe5m7PTmCxAAqnHeogWTOfXJeaBF+kO1UQ== 2024-09-21T01:27:43.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa17407af40 msgr2=0x7fa17407d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:43.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa17407af40 0x7fa17407d450 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa184004dd0 tx=0x7fa184004cd0 comp rx=0 tx=0).stop 2024-09-21T01:27:43.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 msgr2=0x7fa188199b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 0x7fa188199b60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa17800b420 tx=0x7fa17800b8e0 comp rx=0 tx=0).stop 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 shutdown_connections 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa17407af40 0x7fa17407d450 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa188104580 0x7fa188199b60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 --2- 172.21.15.90:0/1953032616 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa188103450 0x7fa1881994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 >> 172.21.15.90:0/1953032616 conn(0x7fa1880fedc0 msgr2=0x7fa188107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 shutdown_connections 2024-09-21T01:27:43.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:27:43.205+0000 7fa1907de640 1 -- 172.21.15.90:0/1953032616 wait complete. 2024-09-21T01:27:43.900 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:43 smithi090 ceph-mon[35142]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:43.900 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:43 smithi090 ceph-mon[35142]: from='client.? 172.21.15.90:0/1953032616' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:27:43.900 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:43 smithi090 ceph-mon[35142]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:27:43.901 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:43 smithi090 ceph-mon[35142]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:27:43.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:43 smithi045 ceph-mon[28361]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:43.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:43 smithi045 ceph-mon[28361]: from='client.? 172.21.15.90:0/1953032616' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:27:43.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:43 smithi045 ceph-mon[28361]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-21T01:27:43.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:43 smithi045 ceph-mon[28361]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T01:27:43.929 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:27:43.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T01:27:43.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T01:27:43.966 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T01:27:44.243 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.690+0000 7f034574e640 1 -- 172.21.15.45:0/1735651924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 msgr2=0x7f0340103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.690+0000 7f034574e640 1 --2- 172.21.15.45:0/1735651924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340103c60 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0330009880 tx=0x7f033002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.690+0000 7f034574e640 1 -- 172.21.15.45:0/1735651924 shutdown_connections 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.690+0000 7f034574e640 1 --2- 172.21.15.45:0/1735651924 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0340109190 0x7f03401095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.690+0000 7f034574e640 1 --2- 172.21.15.45:0/1735651924 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340103c60 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.690+0000 7f034574e640 1 -- 172.21.15.45:0/1735651924 >> 172.21.15.45:0/1735651924 conn(0x7f03400ff2a0 msgr2=0x7f03401016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:44.692 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.691+0000 7f034574e640 1 -- 172.21.15.45:0/1735651924 shutdown_connections 2024-09-21T01:27:44.693 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.691+0000 7f034574e640 1 -- 172.21.15.45:0/1735651924 wait complete. 2024-09-21T01:27:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.691+0000 7f034574e640 1 Processor -- start 2024-09-21T01:27:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f034574e640 1 -- start start 2024-09-21T01:27:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f034574e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f034574e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0340109190 0x7f0340199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f034574e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0340110920 con 0x7f0340103850 2024-09-21T01:27:44.694 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f034574e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f03401107a0 con 0x7f0340109190 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55856/0 (socket says 172.21.15.45:55856) 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033effd640 1 -- 172.21.15.45:0/1018815028 learned_addr learned my addr 172.21.15.45:0/1018815028 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033e7fc640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0340109190 0x7f0340199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033effd640 1 -- 172.21.15.45:0/1018815028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0340109190 msgr2=0x7f0340199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033effd640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0340109190 0x7f0340199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.692+0000 7f033effd640 1 -- 172.21.15.45:0/1018815028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034019f0b0 con 0x7f0340103850 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.693+0000 7f033effd640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340199870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0330006010 tx=0x7f03300045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.693+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f033003f070 con 0x7f0340103850 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.693+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034019f340 con 0x7f0340103850 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.693+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f03300049d0 con 0x7f0340103850 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.693+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0330035030 con 0x7f0340103850 2024-09-21T01:27:44.695 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.693+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034019f8f0 con 0x7f0340103850 2024-09-21T01:27:44.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.694+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0340104ad0 con 0x7f0340103850 2024-09-21T01:27:44.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.695+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f0330042070 con 0x7f0340103850 2024-09-21T01:27:44.696 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.696+0000 7f0327fff640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f032807af80 0x7f032807d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:44.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.696+0000 7f033e7fc640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f032807af80 0x7f032807d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:44.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.696+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f03300be9d0 con 0x7f0340103850 2024-09-21T01:27:44.697 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.696+0000 7f033e7fc640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f032807af80 0x7f032807d490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0334005fd0 tx=0x7f0334005e20 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:44.701 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.700+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f033008a1b0 con 0x7f0340103850 2024-09-21T01:27:44.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.891+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f034019a4f0 con 0x7f0340103850 2024-09-21T01:27:44.893 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.892+0000 7f0327fff640 1 -- 172.21.15.45:0/1018815028 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v42) ==== 86+0+187 (secure 0 0 0) 0x7f033008f060 con 0x7f0340103850 2024-09-21T01:27:44.893 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f032807af80 msgr2=0x7f032807d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f032807af80 0x7f032807d490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0334005fd0 tx=0x7f0334005e20 comp rx=0 tx=0).stop 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 msgr2=0x7f0340199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340199870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0330006010 tx=0x7f03300045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 shutdown_connections 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f032807af80 0x7f032807d490 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0340109190 0x7f0340199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 --2- 172.21.15.45:0/1018815028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0340103850 0x7f0340199870 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:44.896 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 >> 172.21.15.45:0/1018815028 conn(0x7f03400ff2a0 msgr2=0x7f03401016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:44.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 shutdown_connections 2024-09-21T01:27:44.897 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:44.895+0000 7f034574e640 1 -- 172.21.15.45:0/1018815028 wait complete. 2024-09-21T01:27:45.605 INFO:teuthology.orchestra.run.smithi045.stdout:[{"version":42,"timestamp":"2024-09-21T01:27:36.217005+0000","name":"","changes":[{"name":"osd/host:smithi090/osd_memory_target","previous_value":"5212129553","new_value":"3909097164"}]}] 2024-09-21T01:27:45.605 INFO:tasks.ceph_manager:config epoch is 42 2024-09-21T01:27:45.605 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T01:27:45.606 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T01:27:45.606 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T01:27:45.869 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:45.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:45 smithi045 ceph-mon[28361]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:45.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:45 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1018815028' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T01:27:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:45 smithi090 ceph-mon[35142]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:45.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:45 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1018815028' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-21T01:27:46.345 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 -- 172.21.15.45:0/2426626116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04781092e0 msgr2=0x7f04781096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 --2- 172.21.15.45:0/2426626116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04781092e0 0x7f04781096f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0468009880 tx=0x7f046802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 -- 172.21.15.45:0/2426626116 shutdown_connections 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 --2- 172.21.15.45:0/2426626116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f04781092e0 0x7f04781096f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 --2- 172.21.15.45:0/2426626116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0478101980 0x7f0478101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 -- 172.21.15.45:0/2426626116 >> 172.21.15.45:0/2426626116 conn(0x7f0478078830 msgr2=0x7f0478078c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 -- 172.21.15.45:0/2426626116 shutdown_connections 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.344+0000 7f047fb70640 1 -- 172.21.15.45:0/2426626116 wait complete. 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047fb70640 1 Processor -- start 2024-09-21T01:27:46.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047fb70640 1 -- start start 2024-09-21T01:27:46.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047fb70640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 0x7f0478199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047fb70640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04781092e0 0x7f0478199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047fb70640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f0478110950 con 0x7f0478101980 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047fb70640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f04781107d0 con 0x7f04781092e0 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d90c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 0x7f0478199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d90c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 0x7f0478199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55876/0 (socket says 172.21.15.45:55876) 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d90c640 1 -- 172.21.15.45:0/3588464189 learned_addr learned my addr 172.21.15.45:0/3588464189 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d10b640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04781092e0 0x7f0478199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d90c640 1 -- 172.21.15.45:0/3588464189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04781092e0 msgr2=0x7f0478199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d90c640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04781092e0 0x7f0478199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d90c640 1 -- 172.21.15.45:0/3588464189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f047819f280 con 0x7f0478101980 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.345+0000 7f047d10b640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04781092e0 0x7f0478199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.346+0000 7f047d90c640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 0x7f0478199a40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f047400b3d0 tx=0x7f047400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:46.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.346+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0474015020 con 0x7f0478101980 2024-09-21T01:27:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.346+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f047819f570 con 0x7f0478101980 2024-09-21T01:27:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.346+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f0474011ac0 con 0x7f0478101980 2024-09-21T01:27:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.346+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f0474010bc0 con 0x7f0478101980 2024-09-21T01:27:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.346+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f047819fad0 con 0x7f0478101980 2024-09-21T01:27:46.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.347+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0478102910 con 0x7f0478101980 2024-09-21T01:27:46.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.348+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f0474018020 con 0x7f0478101980 2024-09-21T01:27:46.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.348+0000 7f046effd640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f046407af40 0x7f046407d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:46.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.348+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f047409d800 con 0x7f0478101980 2024-09-21T01:27:46.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.348+0000 7f047d10b640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f046407af40 0x7f046407d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:46.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.349+0000 7f047d10b640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f046407af40 0x7f046407d450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f0468000bc0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:46.354 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.353+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f047401f020 con 0x7f0478101980 2024-09-21T01:27:46.600 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.599+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f047819a710 con 0x7f0478101980 2024-09-21T01:27:46.603 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.602+0000 7f046effd640 1 -- 172.21.15.45:0/3588464189 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) ==== 74+0+198157 (secure 0 0 0) 0x7f0474069000 con 0x7f0478101980 2024-09-21T01:27:46.607 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f046407af40 msgr2=0x7f046407d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f046407af40 0x7f046407d450 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0468006010 tx=0x7f0468000bc0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 msgr2=0x7f0478199a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 0x7f0478199a40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f047400b3d0 tx=0x7f047400b890 comp rx=0 tx=0).stop 2024-09-21T01:27:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 shutdown_connections 2024-09-21T01:27:46.609 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f046407af40 0x7f046407d450 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f04781092e0 0x7f0478199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 --2- 172.21.15.45:0/3588464189 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0478101980 0x7f0478199a40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 >> 172.21.15.45:0/3588464189 conn(0x7f0478078830 msgr2=0x7f04781003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:46.610 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 shutdown_connections 2024-09-21T01:27:46.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:46.605+0000 7f047fb70640 1 -- 172.21.15.45:0/3588464189 wait complete. 2024-09-21T01:27:46.918 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:46 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3588464189' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T01:27:46.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:46 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3588464189' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T01:27:47.209 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":19,"flags":0,"active_gid":14227,"active_name":"smithi045.txjudz","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6800","nonce":3847851980},{"type":"v1","addr":"172.21.15.45:6801","nonce":3847851980}]},"active_addr":"172.21.15.45:6801/3847851980","active_change":"2024-09-21T01:22:51.944254+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":14252,"name":"smithi090.zsfvyu","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.45:8443/","prometheus":"http://172.21.15.45:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":4037315678}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":2199498161}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":3163456460}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.45:0","nonce":389379780}]}]} 2024-09-21T01:27:47.214 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T01:27:47.214 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T01:27:47.214 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:27:47.470 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:47.902 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:47 smithi045 ceph-mon[28361]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 -- 172.21.15.45:0/1192660919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 msgr2=0x7fdc3c103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 --2- 172.21.15.45:0/1192660919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c103c60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009880 tx=0x7fdc3802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 -- 172.21.15.45:0/1192660919 shutdown_connections 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 --2- 172.21.15.45:0/1192660919 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 0x7fdc3c1095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 --2- 172.21.15.45:0/1192660919 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c103c60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 -- 172.21.15.45:0/1192660919 >> 172.21.15.45:0/1192660919 conn(0x7fdc3c0ff2a0 msgr2=0x7fdc3c1016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 -- 172.21.15.45:0/1192660919 shutdown_connections 2024-09-21T01:27:47.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.902+0000 7fdc43923640 1 -- 172.21.15.45:0/1192660919 wait complete. 2024-09-21T01:27:47.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc43923640 1 Processor -- start 2024-09-21T01:27:47.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc43923640 1 -- start start 2024-09-21T01:27:47.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc43923640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:47.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc43923640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 0x7fdc3c075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:47.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc43923640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c110900 con 0x7fdc3c103850 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc43923640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc3c110780 con 0x7fdc3c109190 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc416bf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc416bf640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48266/0 (socket says 172.21.15.45:48266) 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc416bf640 1 -- 172.21.15.45:0/841695301 learned_addr learned my addr 172.21.15.45:0/841695301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.903+0000 7fdc40ebe640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 0x7fdc3c075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc416bf640 1 -- 172.21.15.45:0/841695301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 msgr2=0x7fdc3c075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc416bf640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 0x7fdc3c075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc416bf640 1 -- 172.21.15.45:0/841695301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc3c0775f0 con 0x7fdc3c103850 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc40ebe640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 0x7fdc3c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc416bf640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c075710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdc38006010 tx=0x7fdc380045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdc3803f070 con 0x7fdc3c103850 2024-09-21T01:27:47.906 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc3c076250 con 0x7fdc3c103850 2024-09-21T01:27:47.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc380049d0 con 0x7fdc3c103850 2024-09-21T01:27:47.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdc38035030 con 0x7fdc3c103850 2024-09-21T01:27:47.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.904+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3c1a5bd0 con 0x7fdc3c103850 2024-09-21T01:27:47.907 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.905+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc3c104ad0 con 0x7fdc3c103850 2024-09-21T01:27:47.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.906+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fdc38042070 con 0x7fdc3c103850 2024-09-21T01:27:47.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.906+0000 7fdc327fc640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2807af30 0x7fdc2807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:47.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.907+0000 7fdc40ebe640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2807af30 0x7fdc2807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:47.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.907+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fdc380bea70 con 0x7fdc3c103850 2024-09-21T01:27:47.908 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.907+0000 7fdc40ebe640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2807af30 0x7fdc2807d440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c108c90 tx=0x7fdc2c00a5c0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:47.912 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:47.911+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdc3808a1d0 con 0x7fdc3c103850 2024-09-21T01:27:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:47 smithi090 ceph-mon[35142]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.102+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc3c10b930 con 0x7fdc3c103850 2024-09-21T01:27:48.103 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.102+0000 7fdc327fc640 1 -- 172.21.15.45:0/841695301 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+13871 (secure 0 0 0) 0x7fdc3808f080 con 0x7fdc3c103850 2024-09-21T01:27:48.104 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:48.104 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":52,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","created":"2024-09-21T01:20:38.644738+0000","modified":"2024-09-21T01:27:38.613151+0000","last_up_change":"2024-09-21T01:27:36.610150+0000","last_in_change":"2024-09-21T01:27:14.953689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:25:30.425195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20b68c0-eefc-40c0-b952-799c83b7f4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6803","nonce":3771776284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6805","nonce":3771776284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6809","nonce":3771776284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6807","nonce":3771776284}]},"public_addr":"172.21.15.45:6803/3771776284","cluster_addr":"172.21.15.45:6805/3771776284","heartbeat_back_addr":"172.21.15.45:6809/3771776284","heartbeat_front_addr":"172.21.15.45:6807/3771776284","state":["exists","up"]},{"osd":1,"uuid":"b10dd316-1566-4bff-933b-1b96b16a3378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6811","nonce":2203931892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6813","nonce":2203931892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6817","nonce":2203931892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6815","nonce":2203931892}]},"public_addr":"172.21.15.45:6811/2203931892","cluster_addr":"172.21.15.45:6813/2203931892","heartbeat_back_addr":"172.21.15.45:6817/2203931892","heartbeat_front_addr":"172.21.15.45:6815/2203931892","state":["exists","up"]},{"osd":2,"uuid":"80cb32e4-abb7-4cd6-b2af-6e254e3c592b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6819","nonce":2681427309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6821","nonce":2681427309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6825","nonce":2681427309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6823","nonce":2681427309}]},"public_addr":"172.21.15.45:6819/2681427309","cluster_addr":"172.21.15.45:6821/2681427309","heartbeat_back_addr":"172.21.15.45:6825/2681427309","heartbeat_front_addr":"172.21.15.45:6823/2681427309","state":["exists","up"]},{"osd":3,"uuid":"1aaec02f-d282-4505-a860-3d77532911df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6827","nonce":2480895441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6829","nonce":2480895441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6833","nonce":2480895441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6831","nonce":2480895441}]},"public_addr":"172.21.15.45:6827/2480895441","cluster_addr":"172.21.15.45:6829/2480895441","heartbeat_back_addr":"172.21.15.45:6833/2480895441","heartbeat_front_addr":"172.21.15.45:6831/2480895441","state":["exists","up"]},{"osd":4,"uuid":"85e89409-05b7-432d-9044-3b730f4f68b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6801","nonce":2331499915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6803","nonce":2331499915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6807","nonce":2331499915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6805","nonce":2331499915}]},"public_addr":"172.21.15.90:6801/2331499915","cluster_addr":"172.21.15.90:6803/2331499915","heartbeat_back_addr":"172.21.15.90:6807/2331499915","heartbeat_front_addr":"172.21.15.90:6805/2331499915","state":["exists","up"]},{"osd":5,"uuid":"0122a5d6-f32b-4851-851e-aa054f411b75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6809","nonce":1613366233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6811","nonce":1613366233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6815","nonce":1613366233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6813","nonce":1613366233}]},"public_addr":"172.21.15.90:6809/1613366233","cluster_addr":"172.21.15.90:6811/1613366233","heartbeat_back_addr":"172.21.15.90:6815/1613366233","heartbeat_front_addr":"172.21.15.90:6813/1613366233","state":["exists","up"]},{"osd":6,"uuid":"5170fdbb-6229-40f8-b415-de7d628a0490","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6817","nonce":2608695333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6819","nonce":2608695333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6823","nonce":2608695333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6821","nonce":2608695333}]},"public_addr":"172.21.15.90:6817/2608695333","cluster_addr":"172.21.15.90:6819/2608695333","heartbeat_back_addr":"172.21.15.90:6823/2608695333","heartbeat_front_addr":"172.21.15.90:6821/2608695333","state":["exists","up"]},{"osd":7,"uuid":"3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6825","nonce":2703954098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6827","nonce":2703954098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6831","nonce":2703954098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6829","nonce":2703954098}]},"public_addr":"172.21.15.90:6825/2703954098","cluster_addr":"172.21.15.90:6827/2703954098","heartbeat_back_addr":"172.21.15.90:6831/2703954098","heartbeat_front_addr":"172.21.15.90:6829/2703954098","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:24:32.659134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:00.007423+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:27.198145+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:54.790013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:18.251812+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:43.822784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:08.545224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:34.701933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/1147989706":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1797066051":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3851449798":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/3707884113":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3194963614":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6800/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1334988077":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6800/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/1726148947":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/308996383":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6801/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:6800/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:0/1437053407":"2024-09-22T01:21:45.172023+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:27:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2807af30 msgr2=0x7fdc2807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2807af30 0x7fdc2807d440 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c108c90 tx=0x7fdc2c00a5c0 comp rx=0 tx=0).stop 2024-09-21T01:27:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 msgr2=0x7fdc3c075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c075710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdc38006010 tx=0x7fdc380045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 shutdown_connections 2024-09-21T01:27:48.106 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2807af30 0x7fdc2807d440 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:48.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3c109190 0x7fdc3c075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:48.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 --2- 172.21.15.45:0/841695301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc3c103850 0x7fdc3c075710 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:48.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 >> 172.21.15.45:0/841695301 conn(0x7fdc3c0ff2a0 msgr2=0x7fdc3c1016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:48.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 shutdown_connections 2024-09-21T01:27:48.107 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:48.105+0000 7fdc43923640 1 -- 172.21.15.45:0/841695301 wait complete. 2024-09-21T01:27:48.738 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:48 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/841695301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:27:48.768 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T01:27:48.768 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:27:48.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:48 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/841695301' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:27:49.069 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 -- 172.21.15.45:0/1493753324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 msgr2=0x7f8bd8103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 --2- 172.21.15.45:0/1493753324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd8103c60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8009880 tx=0x7f8bc802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 -- 172.21.15.45:0/1493753324 shutdown_connections 2024-09-21T01:27:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 --2- 172.21.15.45:0/1493753324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8109190 0x7f8bd81095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 --2- 172.21.15.45:0/1493753324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd8103c60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 -- 172.21.15.45:0/1493753324 >> 172.21.15.45:0/1493753324 conn(0x7f8bd80ff2a0 msgr2=0x7f8bd81016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 -- 172.21.15.45:0/1493753324 shutdown_connections 2024-09-21T01:27:49.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.491+0000 7f8bde398640 1 -- 172.21.15.45:0/1493753324 wait complete. 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bde398640 1 Processor -- start 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bde398640 1 -- start start 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bde398640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd81998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bde398640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8109190 0x7f8bd8199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bde398640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd8110920 con 0x7f8bd8103850 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bde398640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd81107a0 con 0x7f8bd8109190 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd81998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:49.495 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.492+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd81998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48278/0 (socket says 172.21.15.45:48278) 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd7fff640 1 -- 172.21.15.45:0/157527100 learned_addr learned my addr 172.21.15.45:0/157527100 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd77fe640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8109190 0x7f8bd8199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd7fff640 1 -- 172.21.15.45:0/157527100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8109190 msgr2=0x7f8bd8199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd7fff640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8109190 0x7f8bd8199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd7fff640 1 -- 172.21.15.45:0/157527100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd819f0e0 con 0x7f8bd8103850 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd7fff640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd81998a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8006010 tx=0x7f8bc80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8bc803f070 con 0x7f8bd8103850 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd819f370 con 0x7f8bd8103850 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f8bc80049d0 con 0x7f8bd8103850 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8bc8035030 con 0x7f8bd8103850 2024-09-21T01:27:49.496 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.493+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd819f920 con 0x7f8bd8103850 2024-09-21T01:27:49.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.494+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd8104ad0 con 0x7f8bd8103850 2024-09-21T01:27:49.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.495+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f8bc8042070 con 0x7f8bd8103850 2024-09-21T01:27:49.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.496+0000 7f8bd57fa640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8bc007af30 0x7f8bc007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:49.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.496+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f8bc80be9b0 con 0x7f8bd8103850 2024-09-21T01:27:49.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.496+0000 7f8bd77fe640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8bc007af30 0x7f8bc007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:49.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.496+0000 7f8bd77fe640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8bc007af30 0x7f8bc007d440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8108c90 tx=0x7f8bcc00a250 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:49.502 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.501+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f8bc808a150 con 0x7f8bd8103850 2024-09-21T01:27:49.706 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.705+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8bd8103c60 con 0x7f8bd8103850 2024-09-21T01:27:49.707 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.706+0000 7f8bd57fa640 1 -- 172.21.15.45:0/157527100 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+13871 (secure 0 0 0) 0x7f8bc807fe20 con 0x7f8bd8103850 2024-09-21T01:27:49.707 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:27:49.707 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":52,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","created":"2024-09-21T01:20:38.644738+0000","modified":"2024-09-21T01:27:38.613151+0000","last_up_change":"2024-09-21T01:27:36.610150+0000","last_in_change":"2024-09-21T01:27:14.953689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:25:30.425195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20b68c0-eefc-40c0-b952-799c83b7f4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6803","nonce":3771776284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6805","nonce":3771776284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6809","nonce":3771776284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6807","nonce":3771776284}]},"public_addr":"172.21.15.45:6803/3771776284","cluster_addr":"172.21.15.45:6805/3771776284","heartbeat_back_addr":"172.21.15.45:6809/3771776284","heartbeat_front_addr":"172.21.15.45:6807/3771776284","state":["exists","up"]},{"osd":1,"uuid":"b10dd316-1566-4bff-933b-1b96b16a3378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6811","nonce":2203931892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6813","nonce":2203931892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6817","nonce":2203931892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6815","nonce":2203931892}]},"public_addr":"172.21.15.45:6811/2203931892","cluster_addr":"172.21.15.45:6813/2203931892","heartbeat_back_addr":"172.21.15.45:6817/2203931892","heartbeat_front_addr":"172.21.15.45:6815/2203931892","state":["exists","up"]},{"osd":2,"uuid":"80cb32e4-abb7-4cd6-b2af-6e254e3c592b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6819","nonce":2681427309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6821","nonce":2681427309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6825","nonce":2681427309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6823","nonce":2681427309}]},"public_addr":"172.21.15.45:6819/2681427309","cluster_addr":"172.21.15.45:6821/2681427309","heartbeat_back_addr":"172.21.15.45:6825/2681427309","heartbeat_front_addr":"172.21.15.45:6823/2681427309","state":["exists","up"]},{"osd":3,"uuid":"1aaec02f-d282-4505-a860-3d77532911df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6827","nonce":2480895441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6829","nonce":2480895441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6833","nonce":2480895441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6831","nonce":2480895441}]},"public_addr":"172.21.15.45:6827/2480895441","cluster_addr":"172.21.15.45:6829/2480895441","heartbeat_back_addr":"172.21.15.45:6833/2480895441","heartbeat_front_addr":"172.21.15.45:6831/2480895441","state":["exists","up"]},{"osd":4,"uuid":"85e89409-05b7-432d-9044-3b730f4f68b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6801","nonce":2331499915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6803","nonce":2331499915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6807","nonce":2331499915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6805","nonce":2331499915}]},"public_addr":"172.21.15.90:6801/2331499915","cluster_addr":"172.21.15.90:6803/2331499915","heartbeat_back_addr":"172.21.15.90:6807/2331499915","heartbeat_front_addr":"172.21.15.90:6805/2331499915","state":["exists","up"]},{"osd":5,"uuid":"0122a5d6-f32b-4851-851e-aa054f411b75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6809","nonce":1613366233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6811","nonce":1613366233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6815","nonce":1613366233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6813","nonce":1613366233}]},"public_addr":"172.21.15.90:6809/1613366233","cluster_addr":"172.21.15.90:6811/1613366233","heartbeat_back_addr":"172.21.15.90:6815/1613366233","heartbeat_front_addr":"172.21.15.90:6813/1613366233","state":["exists","up"]},{"osd":6,"uuid":"5170fdbb-6229-40f8-b415-de7d628a0490","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6817","nonce":2608695333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6819","nonce":2608695333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6823","nonce":2608695333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6821","nonce":2608695333}]},"public_addr":"172.21.15.90:6817/2608695333","cluster_addr":"172.21.15.90:6819/2608695333","heartbeat_back_addr":"172.21.15.90:6823/2608695333","heartbeat_front_addr":"172.21.15.90:6821/2608695333","state":["exists","up"]},{"osd":7,"uuid":"3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6825","nonce":2703954098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6827","nonce":2703954098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6831","nonce":2703954098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6829","nonce":2703954098}]},"public_addr":"172.21.15.90:6825/2703954098","cluster_addr":"172.21.15.90:6827/2703954098","heartbeat_back_addr":"172.21.15.90:6831/2703954098","heartbeat_front_addr":"172.21.15.90:6829/2703954098","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:24:32.659134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:00.007423+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:27.198145+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:54.790013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:18.251812+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:43.822784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:08.545224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:34.701933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/1147989706":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1797066051":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3851449798":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/3707884113":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3194963614":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6800/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1334988077":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6800/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/1726148947":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/308996383":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6801/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:6800/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:0/1437053407":"2024-09-22T01:21:45.172023+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:27:49.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8bc007af30 msgr2=0x7f8bc007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8bc007af30 0x7f8bc007d440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8bd8108c90 tx=0x7f8bcc00a250 comp rx=0 tx=0).stop 2024-09-21T01:27:49.709 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 msgr2=0x7f8bd81998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd81998a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8bc8006010 tx=0x7f8bc80045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 shutdown_connections 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f8bc007af30 0x7f8bc007d440 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8bd8109190 0x7f8bd8199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 --2- 172.21.15.45:0/157527100 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8bd8103850 0x7f8bd81998a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 >> 172.21.15.45:0/157527100 conn(0x7f8bd80ff2a0 msgr2=0x7f8bd81016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 shutdown_connections 2024-09-21T01:27:49.710 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:49.708+0000 7f8bde398640 1 -- 172.21.15.45:0/157527100 wait complete. 2024-09-21T01:27:49.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:49 smithi045 ceph-mon[28361]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:49 smithi090 ceph-mon[35142]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:50.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T01:27:50.292 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T01:27:50.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T01:27:50.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T01:27:50.293 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T01:27:50.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T01:27:50.294 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T01:27:50.295 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T01:27:50.660 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:50 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/157527100' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:27:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.825 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.826 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.828 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.830 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.831 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.837 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:50.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:50 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/157527100' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:27:51.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:51 smithi045 ceph-mon[28361]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:51.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:51 smithi090 ceph-mon[35142]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:52.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 -- 172.21.15.45:0/1520169227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 msgr2=0x7f31501084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 --2- 172.21.15.45:0/1520169227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f31501084a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f313c009880 tx=0x7f313c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 -- 172.21.15.45:0/1520169227 shutdown_connections 2024-09-21T01:27:52.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 --2- 172.21.15.45:0/1520169227 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f31501084a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 --2- 172.21.15.45:0/1520169227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3150102750 0x7f3150102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 -- 172.21.15.45:0/1520169227 >> 172.21.15.45:0/1520169227 conn(0x7f31500fe1a0 msgr2=0x7f31501005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 -- 172.21.15.45:0/1520169227 shutdown_connections 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 -- 172.21.15.45:0/1520169227 wait complete. 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.110+0000 7f3156d81640 1 Processor -- start 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f3156d81640 1 -- start start 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f3156d81640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3150102750 0x7f315019a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f3156d81640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f315019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f3156d81640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f315010f7c0 con 0x7f3150108090 2024-09-21T01:27:52.118 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f3156d81640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f315010f640 con 0x7f3150102750 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f314ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f315019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f314ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f315019aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48298/0 (socket says 172.21.15.45:48298) 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f314ffff640 1 -- 172.21.15.45:0/4251663490 learned_addr learned my addr 172.21.15.45:0/4251663490 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f3154b1d640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3150102750 0x7f315019a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f314ffff640 1 -- 172.21.15.45:0/4251663490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3150102750 msgr2=0x7f315019a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f314ffff640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3150102750 0x7f315019a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.111+0000 7f314ffff640 1 -- 172.21.15.45:0/4251663490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315019b0e0 con 0x7f3150108090 2024-09-21T01:27:52.119 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.112+0000 7f314ffff640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f315019aba0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f313c02a9c0 tx=0x7f313c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.112+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f313c03f070 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.112+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f313c004950 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.112+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315019b370 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.112+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f313c035030 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.112+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31501a38a0 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.113+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3150063b10 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.114+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f313c042070 con 0x7f3150108090 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.114+0000 7f314dffb640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f314007af30 0x7f314007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.120 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f3154b1d640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f314007af30 0x7f314007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f313c0bec00 con 0x7f3150108090 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f314dffb640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] conn(0x7f3140081340 0x7f3140083790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 --> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f313c045e50 con 0x7f3140081340 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f313c0bf0a0 con 0x7f3150108090 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f315531e640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] conn(0x7f3140081340 0x7f3140083790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f3154b1d640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f314007af30 0x7f314007d440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f315006b6f0 tx=0x7f314400b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.121 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.115+0000 7f315531e640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] conn(0x7f3140081340 0x7f3140083790 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.122 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.117+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== osd.3 v2:172.21.15.45:6826/2480895441 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f313c045e50 con 0x7f3140081340 2024-09-21T01:27:52.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 --> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3150063f60 con 0x7f3140081340 2024-09-21T01:27:52.132 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f314dffb640 1 -- 172.21.15.45:0/4251663490 <== osd.3 v2:172.21.15.45:6826/2480895441 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3150063f60 con 0x7f3140081340 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] conn(0x7f3140081340 msgr2=0x7f3140083790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] conn(0x7f3140081340 0x7f3140083790 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f314007af30 msgr2=0x7f314007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f314007af30 0x7f314007d440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f315006b6f0 tx=0x7f314400b390 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 msgr2=0x7f315019aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f315019aba0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f313c02a9c0 tx=0x7f313c0045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 shutdown_connections 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6826/2480895441,v1:172.21.15.45:6827/2480895441] conn(0x7f3140081340 0x7f3140083790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f314007af30 0x7f314007d440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3150108090 0x7f315019aba0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 --2- 172.21.15.45:0/4251663490 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3150102750 0x7f315019a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.133 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 >> 172.21.15.45:0/4251663490 conn(0x7f31500fe1a0 msgr2=0x7f31501005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:52.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 shutdown_connections 2024-09-21T01:27:52.134 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.131+0000 7f3156d81640 1 -- 172.21.15.45:0/4251663490 wait complete. 2024-09-21T01:27:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 -- 172.21.15.45:0/1756562835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 msgr2=0x7f2ef81084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 --2- 172.21.15.45:0/1756562835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef81084a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009880 tx=0x7f2eec02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.317 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 -- 172.21.15.45:0/1756562835 shutdown_connections 2024-09-21T01:27:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 --2- 172.21.15.45:0/1756562835 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef81084a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 --2- 172.21.15.45:0/1756562835 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ef8102750 0x7f2ef8102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 -- 172.21.15.45:0/1756562835 >> 172.21.15.45:0/1756562835 conn(0x7f2ef80fe1a0 msgr2=0x7f2ef81005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 -- 172.21.15.45:0/1756562835 shutdown_connections 2024-09-21T01:27:52.318 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.316+0000 7f2efe18c640 1 -- 172.21.15.45:0/1756562835 wait complete. 2024-09-21T01:27:52.319 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2efe18c640 1 Processor -- start 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2efe18c640 1 -- start start 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2efe18c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ef8102750 0x7f2ef819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2efe18c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef819abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2efe18c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef810f7e0 con 0x7f2ef8108090 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2efe18c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f2ef810f660 con 0x7f2ef8102750 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.317+0000 7f2ef6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef819abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef819abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48316/0 (socket says 172.21.15.45:48316) 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef6ffd640 1 -- 172.21.15.45:0/3402338767 learned_addr learned my addr 172.21.15.45:0/3402338767 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef77fe640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ef8102750 0x7f2ef819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef6ffd640 1 -- 172.21.15.45:0/3402338767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ef8102750 msgr2=0x7f2ef819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef6ffd640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ef8102750 0x7f2ef819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.320 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef6ffd640 1 -- 172.21.15.45:0/3402338767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ef819b110 con 0x7f2ef8108090 2024-09-21T01:27:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef6ffd640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef819abd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2eec02a9c0 tx=0x7f2eec0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2eec03f070 con 0x7f2ef8108090 2024-09-21T01:27:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f2eec004950 con 0x7f2ef8108090 2024-09-21T01:27:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ef819b3a0 con 0x7f2ef8108090 2024-09-21T01:27:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f2eec035030 con 0x7f2ef8108090 2024-09-21T01:27:52.321 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.318+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ef81a38d0 con 0x7f2ef8108090 2024-09-21T01:27:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.319+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ef8063b10 con 0x7f2ef8108090 2024-09-21T01:27:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.320+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f2eec042070 con 0x7f2ef8108090 2024-09-21T01:27:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef4ff9640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2ee007af80 0x7f2ee007d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f2eec0be740 con 0x7f2ef8108090 2024-09-21T01:27:52.322 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef77fe640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2ee007af80 0x7f2ee007d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef4ff9640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] conn(0x7f2ee0081390 0x7f2ee00837e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 --> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2eec037ec0 con 0x7f2ee0081390 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f2eec0c32a0 con 0x7f2ef8108090 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef7fff640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] conn(0x7f2ee0081390 0x7f2ee00837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.321+0000 7f2ef77fe640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2ee007af80 0x7f2ee007d490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ef806b6f0 tx=0x7f2ee8009680 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.322+0000 7f2ef7fff640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] conn(0x7f2ee0081390 0x7f2ee00837e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.323 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.322+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== osd.0 v2:172.21.15.45:6802/3771776284 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2eec037ec0 con 0x7f2ee0081390 2024-09-21T01:27:52.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 --> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ef8063f90 con 0x7f2ee0081390 2024-09-21T01:27:52.337 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2ef4ff9640 1 -- 172.21.15.45:0/3402338767 <== osd.0 v2:172.21.15.45:6802/3771776284 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2ef8063f90 con 0x7f2ee0081390 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] conn(0x7f2ee0081390 msgr2=0x7f2ee00837e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] conn(0x7f2ee0081390 0x7f2ee00837e0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2ee007af80 msgr2=0x7f2ee007d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2ee007af80 0x7f2ee007d490 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2ef806b6f0 tx=0x7f2ee8009680 comp rx=0 tx=0).stop 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 msgr2=0x7f2ef819abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef819abd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2eec02a9c0 tx=0x7f2eec0045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 shutdown_connections 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6802/3771776284,v1:172.21.15.45:6803/3771776284] conn(0x7f2ee0081390 0x7f2ee00837e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.338 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f2ee007af80 0x7f2ee007d490 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2ef8108090 0x7f2ef819abd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 --2- 172.21.15.45:0/3402338767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ef8102750 0x7f2ef819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.336+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 >> 172.21.15.45:0/3402338767 conn(0x7f2ef80fe1a0 msgr2=0x7f2ef81005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:52.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.337+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 shutdown_connections 2024-09-21T01:27:52.339 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.337+0000 7f2efe18c640 1 -- 172.21.15.45:0/3402338767 wait complete. 2024-09-21T01:27:52.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.819+0000 7f836116c640 1 -- 172.21.15.45:0/3595761671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c108090 msgr2=0x7f835c1084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.819+0000 7f836116c640 1 --2- 172.21.15.45:0/3595761671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c108090 0x7f835c1084a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8350009880 tx=0x7f835002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 -- 172.21.15.45:0/3595761671 shutdown_connections 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 --2- 172.21.15.45:0/3595761671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c108090 0x7f835c1084a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 --2- 172.21.15.45:0/3595761671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c102750 0x7f835c102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 -- 172.21.15.45:0/3595761671 >> 172.21.15.45:0/3595761671 conn(0x7f835c0fe1a0 msgr2=0x7f835c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 -- 172.21.15.45:0/3595761671 shutdown_connections 2024-09-21T01:27:52.821 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 -- 172.21.15.45:0/3595761671 wait complete. 2024-09-21T01:27:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.820+0000 7f836116c640 1 Processor -- start 2024-09-21T01:27:52.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f836116c640 1 -- start start 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f836116c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 0x7f835c198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f836116c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c108090 0x7f835c198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f836116c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f835c10f7c0 con 0x7f835c102750 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f836116c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f835c10f640 con 0x7f835c108090 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 0x7f835c198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835ad9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 0x7f835c198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48326/0 (socket says 172.21.15.45:48326) 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835ad9d640 1 -- 172.21.15.45:0/2998859655 learned_addr learned my addr 172.21.15.45:0/2998859655 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835a59c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c108090 0x7f835c198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835ad9d640 1 -- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c108090 msgr2=0x7f835c198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835ad9d640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c108090 0x7f835c198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835ad9d640 1 -- 172.21.15.45:0/2998859655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835c06b990 con 0x7f835c102750 2024-09-21T01:27:52.824 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.821+0000 7f835a59c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c108090 0x7f835c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:27:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.822+0000 7f835ad9d640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 0x7f835c198450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f834c00cc60 tx=0x7f834c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.822+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f834c00ce10 con 0x7f835c102750 2024-09-21T01:27:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.822+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835c198ed0 con 0x7f835c102750 2024-09-21T01:27:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.822+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f834c00f8e0 con 0x7f835c102750 2024-09-21T01:27:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.822+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f834c0188b0 con 0x7f835c102750 2024-09-21T01:27:52.825 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.822+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835c1a5b40 con 0x7f835c102750 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.823+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f835c063b10 con 0x7f835c102750 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.824+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f834c018a70 con 0x7f835c102750 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.824+0000 7f8343fff640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f834407af30 0x7f834407d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.824+0000 7f835a59c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f834407af30 0x7f834407d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.824+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f834c09e400 con 0x7f835c102750 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.825+0000 7f8343fff640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] conn(0x7f8344081340 0x7f8344083790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.825+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 --> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8344083e60 con 0x7f8344081340 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.825+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f834c0a5050 con 0x7f835c102750 2024-09-21T01:27:52.826 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.825+0000 7f835a59c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f834407af30 0x7f834407d440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8350000c00 tx=0x7f8350004220 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.825+0000 7f835b59e640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] conn(0x7f8344081340 0x7f8344083790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.826+0000 7f835b59e640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] conn(0x7f8344081340 0x7f8344083790 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:52.827 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.826+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== osd.5 v2:172.21.15.90:6808/1613366233 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8344083e60 con 0x7f8344081340 2024-09-21T01:27:52.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.840+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 --> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f835c063f90 con 0x7f8344081340 2024-09-21T01:27:52.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.840+0000 7f8343fff640 1 -- 172.21.15.45:0/2998859655 <== osd.5 v2:172.21.15.90:6808/1613366233 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f835c063f90 con 0x7f8344081340 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] conn(0x7f8344081340 msgr2=0x7f8344083790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] conn(0x7f8344081340 0x7f8344083790 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f834407af30 msgr2=0x7f834407d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f834407af30 0x7f834407d440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8350000c00 tx=0x7f8350004220 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 msgr2=0x7f835c198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 0x7f835c198450 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f834c00cc60 tx=0x7f834c00a620 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 shutdown_connections 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:6808/1613366233,v1:172.21.15.90:6809/1613366233] conn(0x7f8344081340 0x7f8344083790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f834407af30 0x7f834407d440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f835c108090 0x7f835c198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 --2- 172.21.15.45:0/2998859655 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f835c102750 0x7f835c198450 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:52.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 >> 172.21.15.45:0/2998859655 conn(0x7f835c0fe1a0 msgr2=0x7f835c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:52.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 shutdown_connections 2024-09-21T01:27:52.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:52.841+0000 7f836116c640 1 -- 172.21.15.45:0/2998859655 wait complete. 2024-09-21T01:27:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:27:53.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:53 smithi045 ceph-mon[28361]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 -- 172.21.15.45:0/4283856138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30108090 msgr2=0x7f3e301084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 --2- 172.21.15.45:0/4283856138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30108090 0x7f3e301084a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009880 tx=0x7f3e2402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 -- 172.21.15.45:0/4283856138 shutdown_connections 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 --2- 172.21.15.45:0/4283856138 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30108090 0x7f3e301084a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 --2- 172.21.15.45:0/4283856138 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30102750 0x7f3e30102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 -- 172.21.15.45:0/4283856138 >> 172.21.15.45:0/4283856138 conn(0x7f3e300fe1a0 msgr2=0x7f3e301005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 -- 172.21.15.45:0/4283856138 shutdown_connections 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.967+0000 7f3e3636e640 1 -- 172.21.15.45:0/4283856138 wait complete. 2024-09-21T01:27:53.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e3636e640 1 Processor -- start 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e3636e640 1 -- start start 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e3636e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 0x7f3e301998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e3636e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30108090 0x7f3e30199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e3636e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3010f7e0 con 0x7f3e30102750 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e3636e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3e3010f660 con 0x7f3e30108090 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 0x7f3e301998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 0x7f3e301998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48396/0 (socket says 172.21.15.45:48396) 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.968+0000 7f3e2ffff640 1 -- 172.21.15.45:0/121506543 learned_addr learned my addr 172.21.15.45:0/121506543 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2f7fe640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30108090 0x7f3e30199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2ffff640 1 -- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30108090 msgr2=0x7f3e30199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2ffff640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30108090 0x7f3e30199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2ffff640 1 -- 172.21.15.45:0/121506543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e3019f110 con 0x7f3e30102750 2024-09-21T01:27:53.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2ffff640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 0x7f3e301998d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000c8f0 tx=0x7f3e2000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3e200079b0 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3e20005020 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3e20004c50 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e3019f400 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.969+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e3019f960 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.970+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e3019a550 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.971+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f3e200051d0 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2d7fa640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3e1807af30 0x7f3e1807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f3e2009e0c0 con 0x7f3e30102750 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2f7fe640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3e1807af30 0x7f3e1807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2d7fa640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] conn(0x7f3e18081340 0x7f3e18083790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:53.989 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 --> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e2001ae50 con 0x7f3e18081340 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f3e2009e560 con 0x7f3e30102750 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e2f7fe640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3e1807af30 0x7f3e1807d440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009230 tx=0x7f3e240058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.972+0000 7f3e3490b640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] conn(0x7f3e18081340 0x7f3e18083790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.973+0000 7f3e3490b640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] conn(0x7f3e18081340 0x7f3e18083790 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.974+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== osd.4 v2:172.21.15.90:6800/2331499915 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e2001ae50 con 0x7f3e18081340 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 --> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3e30104300 con 0x7f3e18081340 2024-09-21T01:27:53.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e2d7fa640 1 -- 172.21.15.45:0/121506543 <== osd.4 v2:172.21.15.90:6800/2331499915 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3e30104300 con 0x7f3e18081340 2024-09-21T01:27:53.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] conn(0x7f3e18081340 msgr2=0x7f3e18083790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] conn(0x7f3e18081340 0x7f3e18083790 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3e1807af30 msgr2=0x7f3e1807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3e1807af30 0x7f3e1807d440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3e24009230 tx=0x7f3e240058d0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 msgr2=0x7f3e301998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.988+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 0x7f3e301998d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000c8f0 tx=0x7f3e2000cdb0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 shutdown_connections 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:6800/2331499915,v1:172.21.15.90:6801/2331499915] conn(0x7f3e18081340 0x7f3e18083790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3e1807af30 0x7f3e1807d440 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e30108090 0x7f3e30199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 --2- 172.21.15.45:0/121506543 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3e30102750 0x7f3e301998d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 >> 172.21.15.45:0/121506543 conn(0x7f3e300fe1a0 msgr2=0x7f3e30101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 shutdown_connections 2024-09-21T01:27:53.992 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.989+0000 7f3e3636e640 1 -- 172.21.15.45:0/121506543 wait complete. 2024-09-21T01:27:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:53 smithi090 ceph-mon[35142]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 -- 172.21.15.45:0/4234881308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 msgr2=0x7f72081095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 --2- 172.21.15.45:0/4234881308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f72081095d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009880 tx=0x7f71f802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 -- 172.21.15.45:0/4234881308 shutdown_connections 2024-09-21T01:27:54.087 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 --2- 172.21.15.45:0/4234881308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f72081095d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 --2- 172.21.15.45:0/4234881308 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 0x7f7208102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 -- 172.21.15.45:0/4234881308 >> 172.21.15.45:0/4234881308 conn(0x7f7208077860 msgr2=0x7f7208077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 -- 172.21.15.45:0/4234881308 shutdown_connections 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.851+0000 7f720f9a2640 1 -- 172.21.15.45:0/4234881308 wait complete. 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720f9a2640 1 Processor -- start 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720f9a2640 1 -- start start 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720f9a2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 0x7f7208102c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720f9a2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f7208103160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720f9a2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f7208110910 con 0x7f72081091c0 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720f9a2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f7208110790 con 0x7f7208101c80 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720cf3d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f7208103160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720cf3d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f7208103160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48374/0 (socket says 172.21.15.45:48374) 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.852+0000 7f720cf3d640 1 -- 172.21.15.45:0/1014480055 learned_addr learned my addr 172.21.15.45:0/1014480055 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:54.088 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720d73e640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 0x7f7208102c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720cf3d640 1 -- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 msgr2=0x7f7208102c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720cf3d640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 0x7f7208102c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720cf3d640 1 -- 172.21.15.45:0/1014480055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7208104b00 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720d73e640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 0x7f7208102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720cf3d640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f7208103160 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f71f802a9c0 tx=0x7f71f80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f71f803f070 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f71f8004950 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7208103760 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f71f8035030 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.853+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72081a5c90 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.854+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7208063ad0 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.855+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f71f8042070 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.855+0000 7f71fe7fc640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f71f407af30 0x7f71f407d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f71f80bed70 con 0x7f72081091c0 2024-09-21T01:27:54.089 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f720d73e640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f71f407af30 0x7f71f407d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f71fe7fc640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] conn(0x7f71f4081340 0x7f71f4083790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 --> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71f8045e50 con 0x7f71f4081340 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f71f80bf170 con 0x7f72081091c0 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f720df3f640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] conn(0x7f71f4081340 0x7f71f4083790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.856+0000 7f720d73e640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f71f407af30 0x7f71f407d440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7208104670 tx=0x7f7204005e70 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.857+0000 7f720df3f640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] conn(0x7f71f4081340 0x7f71f4083790 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.857+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== osd.7 v2:172.21.15.90:6824/2703954098 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f71f8045e50 con 0x7f71f4081340 2024-09-21T01:27:54.090 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.871+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 --> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7208063f20 con 0x7f71f4081340 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f71fe7fc640 1 -- 172.21.15.45:0/1014480055 <== osd.7 v2:172.21.15.90:6824/2703954098 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7208063f20 con 0x7f71f4081340 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] conn(0x7f71f4081340 msgr2=0x7f71f4083790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] conn(0x7f71f4081340 0x7f71f4083790 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f71f407af30 msgr2=0x7f71f407d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f71f407af30 0x7f71f407d440 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7208104670 tx=0x7f7204005e70 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 msgr2=0x7f7208103160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f7208103160 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f71f802a9c0 tx=0x7f71f80045e0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 shutdown_connections 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:6824/2703954098,v1:172.21.15.90:6825/2703954098] conn(0x7f71f4081340 0x7f71f4083790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f71f407af30 0x7f71f407d440 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f72081091c0 0x7f7208103160 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 --2- 172.21.15.45:0/1014480055 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7208101c80 0x7f7208102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.091 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 >> 172.21.15.45:0/1014480055 conn(0x7f7208077860 msgr2=0x7f72081052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:54.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 shutdown_connections 2024-09-21T01:27:54.092 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.872+0000 7f720f9a2640 1 -- 172.21.15.45:0/1014480055 wait complete. 2024-09-21T01:27:54.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 -- 172.21.15.45:0/1263312329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 msgr2=0x7f709c10d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.346 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/1263312329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c10d540 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f708c009880 tx=0x7f708c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 -- 172.21.15.45:0/1263312329 shutdown_connections 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/1263312329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c10d540 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/1263312329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f709c1046a0 0x7f709c104ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 -- 172.21.15.45:0/1263312329 >> 172.21.15.45:0/1263312329 conn(0x7f709c0fe1a0 msgr2=0x7f709c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 -- 172.21.15.45:0/1263312329 shutdown_connections 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.021+0000 7f70a3ea2640 1 -- 172.21.15.45:0/1263312329 wait complete. 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a3ea2640 1 Processor -- start 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a3ea2640 1 -- start start 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a3ea2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f709c1046a0 0x7f709c199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.347 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a3ea2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a3ea2640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f709c1108b0 con 0x7f709c104ff0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a3ea2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f709c110730 con 0x7f709c1046a0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48352/0 (socket says 172.21.15.45:48352) 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 -- 172.21.15.45:0/746296356 learned_addr learned my addr 172.21.15.45:0/746296356 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a1c3e640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f709c1046a0 0x7f709c199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 -- 172.21.15.45:0/746296356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f709c1046a0 msgr2=0x7f709c199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f709c1046a0 0x7f709c199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 -- 172.21.15.45:0/746296356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f709c19f0a0 con 0x7f709c104ff0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.022+0000 7f70a143d640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c199e00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f708c02a9c0 tx=0x7f708c004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.023+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f708c03f070 con 0x7f709c104ff0 2024-09-21T01:27:54.348 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.023+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f708c005080 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.023+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f709c19f330 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.023+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f708c035030 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.023+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f709c19f8e0 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.023+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f709c19a4e0 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f708c042070 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f7092ffd640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f708807af60 0x7f708807d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f708c0bebd0 con 0x7f709c104ff0 2024-09-21T01:27:54.349 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f70a1c3e640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f708807af60 0x7f708807d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f7092ffd640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] conn(0x7f7088081370 0x7f70880837c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:54.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 --> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f708c04de50 con 0x7f7088081370 2024-09-21T01:27:54.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f708c0bf070 con 0x7f709c104ff0 2024-09-21T01:27:54.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.025+0000 7f70a243f640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] conn(0x7f7088081370 0x7f70880837c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:54.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.026+0000 7f70a1c3e640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f708807af60 0x7f708807d470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f709800bbe0 tx=0x7f709800b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:54.350 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.026+0000 7f70a243f640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] conn(0x7f7088081370 0x7f70880837c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.026+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== osd.2 v2:172.21.15.45:6818/2681427309 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f708c04de50 con 0x7f7088081370 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.040+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 --> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f709c108e50 con 0x7f7088081370 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.040+0000 7f7092ffd640 1 -- 172.21.15.45:0/746296356 <== osd.2 v2:172.21.15.45:6818/2681427309 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f709c108e50 con 0x7f7088081370 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.040+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] conn(0x7f7088081370 msgr2=0x7f70880837c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] conn(0x7f7088081370 0x7f70880837c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f708807af60 msgr2=0x7f708807d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f708807af60 0x7f708807d470 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f709800bbe0 tx=0x7f709800b390 comp rx=0 tx=0).stop 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 msgr2=0x7f709c199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c199e00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f708c02a9c0 tx=0x7f708c004c60 comp rx=0 tx=0).stop 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 shutdown_connections 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6818/2681427309,v1:172.21.15.45:6819/2681427309] conn(0x7f7088081370 0x7f70880837c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f708807af60 0x7f708807d470 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f709c104ff0 0x7f709c199e00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.351 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 --2- 172.21.15.45:0/746296356 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f709c1046a0 0x7f709c199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:54.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 >> 172.21.15.45:0/746296356 conn(0x7f709c0fe1a0 msgr2=0x7f709c10ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:54.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 shutdown_connections 2024-09-21T01:27:54.352 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:53.041+0000 7f70a3ea2640 1 -- 172.21.15.45:0/746296356 wait complete. 2024-09-21T01:27:55.178 INFO:teuthology.orchestra.run.smithi045.stdout:115964117018 2024-09-21T01:27:55.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 -- 172.21.15.45:0/1109646689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c105020 msgr2=0x7fcc4c10d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 --2- 172.21.15.45:0/1109646689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c105020 0x7fcc4c10d570 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009880 tx=0x7fcc4002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 -- 172.21.15.45:0/1109646689 shutdown_connections 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 --2- 172.21.15.45:0/1109646689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c105020 0x7fcc4c10d570 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 --2- 172.21.15.45:0/1109646689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 -- 172.21.15.45:0/1109646689 >> 172.21.15.45:0/1109646689 conn(0x7fcc4c0fe1d0 msgr2=0x7fcc4c100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 -- 172.21.15.45:0/1109646689 shutdown_connections 2024-09-21T01:27:55.611 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.594+0000 7fcc52390640 1 -- 172.21.15.45:0/1109646689 wait complete. 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.595+0000 7fcc52390640 1 Processor -- start 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.595+0000 7fcc52390640 1 -- start start 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc52390640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc52390640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc4c105020 0x7fcc4c102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc52390640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c110960 con 0x7fcc4c1046d0 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc52390640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcc4c1107e0 con 0x7fcc4c105020 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c102410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48410/0 (socket says 172.21.15.45:48410) 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 -- 172.21.15.45:0/400441754 learned_addr learned my addr 172.21.15.45:0/400441754 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 -- 172.21.15.45:0/400441754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc4c105020 msgr2=0x7fcc4c102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc4c105020 0x7fcc4c102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 -- 172.21.15.45:0/400441754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc4c1042f0 con 0x7fcc4c1046d0 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc4bfff640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c102410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00b420 tx=0x7fcc3c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:55.612 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcc3c011630 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcc3c016070 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc4c102fb0 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcc3c010b80 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.596+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc4c1a5cd0 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.597+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcc4c063ad0 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.598+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fcc3c0117d0 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc497fa640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcc3407af60 0x7fcc3407d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fcc3c09e130 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc4b7fe640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcc3407af60 0x7fcc3407d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc497fa640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] conn(0x7fcc34081370 0x7fcc340837c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 --> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc3c01ee50 con 0x7fcc34081370 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fcc3c09e530 con 0x7fcc4c1046d0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc5092d640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] conn(0x7fcc34081370 0x7fcc340837c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:55.613 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.599+0000 7fcc4b7fe640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcc3407af60 0x7fcc3407d470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcc4002a9c0 tx=0x7fcc40000bc0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:55.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.600+0000 7fcc5092d640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] conn(0x7fcc34081370 0x7fcc340837c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:55.614 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.600+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== osd.1 v2:172.21.15.45:6810/2203931892 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcc3c01ee50 con 0x7fcc34081370 2024-09-21T01:27:55.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.614+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 --> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcc4c063f20 con 0x7fcc34081370 2024-09-21T01:27:55.615 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.614+0000 7fcc497fa640 1 -- 172.21.15.45:0/400441754 <== osd.1 v2:172.21.15.45:6810/2203931892 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcc4c063f20 con 0x7fcc34081370 2024-09-21T01:27:55.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] conn(0x7fcc34081370 msgr2=0x7fcc340837c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.616 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] conn(0x7fcc34081370 0x7fcc340837c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcc3407af60 msgr2=0x7fcc3407d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcc3407af60 0x7fcc3407d470 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcc4002a9c0 tx=0x7fcc40000bc0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 msgr2=0x7fcc4c102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c102410 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00b420 tx=0x7fcc3c00b8e0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 shutdown_connections 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6810/2203931892,v1:172.21.15.45:6811/2203931892] conn(0x7fcc34081370 0x7fcc340837c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcc3407af60 0x7fcc3407d470 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc4c105020 0x7fcc4c102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 --2- 172.21.15.45:0/400441754 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcc4c1046d0 0x7fcc4c102410 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 >> 172.21.15.45:0/400441754 conn(0x7fcc4c0fe1d0 msgr2=0x7fcc4c107ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:55.617 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 shutdown_connections 2024-09-21T01:27:55.618 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:55.615+0000 7fcc52390640 1 -- 172.21.15.45:0/400441754 wait complete. 2024-09-21T01:27:55.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:55 smithi045 ceph-mon[28361]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:55 smithi090 ceph-mon[35142]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:56.409 INFO:teuthology.orchestra.run.smithi045.stdout:137438953493 2024-09-21T01:27:56.409 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T01:27:56.795 INFO:teuthology.orchestra.run.smithi045.stdout:214748364805 2024-09-21T01:27:56.796 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T01:27:57.449 INFO:teuthology.orchestra.run.smithi045.stdout:38654705707 2024-09-21T01:27:57.450 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T01:27:57.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.647+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/3749464564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 msgr2=0x7fdc34105e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.647+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/3749464564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc34105e70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009c20 tx=0x7fdc2c02a970 comp rx=0 tx=0).stop 2024-09-21T01:27:57.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.648+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/3749464564 shutdown_connections 2024-09-21T01:27:57.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.648+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/3749464564 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3410b3a0 0x7fdc3410b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.648+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/3749464564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc34105e70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.648+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/3749464564 >> 172.21.15.45:0/3749464564 conn(0x7fdc34101510 msgr2=0x7fdc34103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.649+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/3749464564 shutdown_connections 2024-09-21T01:27:57.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.649+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/3749464564 wait complete. 2024-09-21T01:27:57.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.650+0000 7fdc3a2c9640 1 Processor -- start 2024-09-21T01:27:57.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.650+0000 7fdc3a2c9640 1 -- start start 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.650+0000 7fdc3a2c9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc3419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.650+0000 7fdc3a2c9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3410b3a0 0x7fdc3419c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.650+0000 7fdc3a2c9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc34112b10 con 0x7fdc34105a60 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.650+0000 7fdc3a2c9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc34112990 con 0x7fdc3410b3a0 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc3419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc3419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58590/0 (socket says 172.21.15.45:58590) 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 -- 172.21.15.45:0/4118140258 learned_addr learned my addr 172.21.15.45:0/4118140258 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:27:57.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc337fe640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3410b3a0 0x7fdc3419c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 -- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3410b3a0 msgr2=0x7fdc3419c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3410b3a0 0x7fdc3419c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 -- 172.21.15.45:0/4118140258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc341a1300 con 0x7fdc34105a60 2024-09-21T01:27:57.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.651+0000 7fdc33fff640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc3419bac0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005e30 tx=0x7fdc2c030e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.652+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdc2c047070 con 0x7fdc34105a60 2024-09-21T01:27:57.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.652+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc341a1590 con 0x7fdc34105a60 2024-09-21T01:27:57.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.652+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc2c02fe40 con 0x7fdc34105a60 2024-09-21T01:27:57.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.652+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc341a1b40 con 0x7fdc34105a60 2024-09-21T01:27:57.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.653+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc3419c740 con 0x7fdc34105a60 2024-09-21T01:27:57.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.652+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdc2c034070 con 0x7fdc34105a60 2024-09-21T01:27:57.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.654+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fdc2c04a070 con 0x7fdc34105a60 2024-09-21T01:27:57.656 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.655+0000 7fdc317fa640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2007af40 0x7fdc2007d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.655+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fdc2c0bf6b0 con 0x7fdc34105a60 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.655+0000 7fdc337fe640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2007af40 0x7fdc2007d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.656+0000 7fdc317fa640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] conn(0x7fdc20081350 0x7fdc200837a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.656+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 --> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc2c041ec0 con 0x7fdc20081350 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.656+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fdc2c0bfb50 con 0x7fdc34105a60 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.656+0000 7fdc337fe640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2007af40 0x7fdc2007d450 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc34107af0 tx=0x7fdc1c00b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.657 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.656+0000 7fdc38866640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] conn(0x7fdc20081350 0x7fdc200837a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:27:57.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.656+0000 7fdc38866640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] conn(0x7fdc20081350 0x7fdc200837a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:27:57.658 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.657+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== osd.6 v2:172.21.15.90:6816/2608695333 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc2c041ec0 con 0x7fdc20081350 2024-09-21T01:27:57.672 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.672+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 --> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc34063a50 con 0x7fdc20081350 2024-09-21T01:27:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.672+0000 7fdc317fa640 1 -- 172.21.15.45:0/4118140258 <== osd.6 v2:172.21.15.90:6816/2608695333 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fdc34063a50 con 0x7fdc20081350 2024-09-21T01:27:57.673 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.672+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] conn(0x7fdc20081350 msgr2=0x7fdc200837a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.672+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] conn(0x7fdc20081350 0x7fdc200837a0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2007af40 msgr2=0x7fdc2007d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2007af40 0x7fdc2007d450 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc34107af0 tx=0x7fdc1c00b390 comp rx=0 tx=0).stop 2024-09-21T01:27:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 msgr2=0x7fdc3419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:27:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc3419bac0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c005e30 tx=0x7fdc2c030e80 comp rx=0 tx=0).stop 2024-09-21T01:27:57.674 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 shutdown_connections 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:6816/2608695333,v1:172.21.15.90:6817/2608695333] conn(0x7fdc20081350 0x7fdc200837a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc2007af40 0x7fdc2007d450 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc3410b3a0 0x7fdc3419c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 --2- 172.21.15.45:0/4118140258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc34105a60 0x7fdc3419bac0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 >> 172.21.15.45:0/4118140258 conn(0x7fdc34101510 msgr2=0x7fdc34103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 shutdown_connections 2024-09-21T01:27:57.675 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:57.673+0000 7fdc3a2c9640 1 -- 172.21.15.45:0/4118140258 wait complete. 2024-09-21T01:27:57.778 INFO:teuthology.orchestra.run.smithi045.stdout:81604378656 2024-09-21T01:27:57.778 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T01:27:57.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:57 smithi045 ceph-mon[28361]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:57 smithi090 ceph-mon[35142]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:58.372 INFO:teuthology.orchestra.run.smithi045.stdout:163208757264 2024-09-21T01:27:58.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T01:27:58.854 INFO:teuthology.orchestra.run.smithi045.stdout:60129542182 2024-09-21T01:27:58.855 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T01:27:59.254 INFO:teuthology.orchestra.run.smithi045.stdout:188978561036 2024-09-21T01:27:59.254 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T01:27:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:59.297 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:59.378 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:59.379 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:27:59.790 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:27:59 smithi045 ceph-mon[28361]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:59.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:27:59 smithi090 ceph-mon[35142]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:27:59.999 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1171725571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 msgr2=0x7fc2b41095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1171725571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b41095a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4009880 tx=0x7fc2a402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1171725571 shutdown_connections 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1171725571 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b41095a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1171725571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 0x7fc2b4103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1171725571 >> 172.21.15.45:0/1171725571 conn(0x7fc2b40ff2a0 msgr2=0x7fc2b41016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1171725571 shutdown_connections 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1171725571 wait complete. 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.998+0000 7fc2b9b16640 1 Processor -- start 2024-09-21T01:28:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b9b16640 1 -- start start 2024-09-21T01:28:00.001 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b9b16640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 0x7fc2b4199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b9b16640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b4199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b9b16640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b41108c0 con 0x7fc2b4109190 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b9b16640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4110740 con 0x7fc2b4103850 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b4199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b4199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58602/0 (socket says 172.21.15.45:58602) 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b2ffd640 1 -- 172.21.15.45:0/1764346097 learned_addr learned my addr 172.21.15.45:0/1764346097 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b37fe640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 0x7fc2b4199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b2ffd640 1 -- 172.21.15.45:0/1764346097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 msgr2=0x7fc2b4199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b2ffd640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 0x7fc2b4199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:27:59.999+0000 7fc2b2ffd640 1 -- 172.21.15.45:0/1764346097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b419f090 con 0x7fc2b4109190 2024-09-21T01:28:00.002 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b37fe640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 0x7fc2b4199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:28:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b2ffd640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b4199df0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc2a402a9c0 tx=0x7fc2a40045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc2a403f070 con 0x7fc2b4109190 2024-09-21T01:28:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fc2a40049d0 con 0x7fc2b4109190 2024-09-21T01:28:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fc2a4035030 con 0x7fc2b4109190 2024-09-21T01:28:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b419f320 con 0x7fc2b4109190 2024-09-21T01:28:00.003 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.000+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b419f8d0 con 0x7fc2b4109190 2024-09-21T01:28:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.001+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b4104ad0 con 0x7fc2b4109190 2024-09-21T01:28:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.002+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fc2a4042070 con 0x7fc2b4109190 2024-09-21T01:28:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.002+0000 7fc2b0ff9640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc2a007af80 0x7fc2a007d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.002+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fc2a40be860 con 0x7fc2b4109190 2024-09-21T01:28:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.002+0000 7fc2b37fe640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc2a007af80 0x7fc2a007d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:00.004 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.003+0000 7fc2b37fe640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc2a007af80 0x7fc2a007d490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc29c00abe0 tx=0x7fc29c00a5c0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:00.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.007+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fc2a408a0e0 con 0x7fc2b4109190 2024-09-21T01:28:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.188+0000 7fcdb3367640 1 -- 172.21.15.45:0/1367830627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 msgr2=0x7fcdac102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.188+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1367830627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac102b60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcda8009880 tx=0x7fcda802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 -- 172.21.15.45:0/1367830627 shutdown_connections 2024-09-21T01:28:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1367830627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 0x7fcdac1084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1367830627 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac102b60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.190 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 -- 172.21.15.45:0/1367830627 >> 172.21.15.45:0/1367830627 conn(0x7fcdac0fe1a0 msgr2=0x7fcdac1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:00.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 -- 172.21.15.45:0/1367830627 shutdown_connections 2024-09-21T01:28:00.191 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 -- 172.21.15.45:0/1367830627 wait complete. 2024-09-21T01:28:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.189+0000 7fcdb3367640 1 Processor -- start 2024-09-21T01:28:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb3367640 1 -- start start 2024-09-21T01:28:00.192 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb3367640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac0fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb3367640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 0x7fcdac0fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb3367640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac10f820 con 0x7fcdac102750 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb3367640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac10f6a0 con 0x7fcdac108090 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac0fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac0fe6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58620/0 (socket says 172.21.15.45:58620) 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 -- 172.21.15.45:0/1765492298 learned_addr learned my addr 172.21.15.45:0/1765492298 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb0902640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 0x7fcdac0fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 -- 172.21.15.45:0/1765492298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 msgr2=0x7fcdac0fec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 0x7fcdac0fec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 -- 172.21.15.45:0/1765492298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac1005b0 con 0x7fcdac102750 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb0902640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 0x7fcdac0fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.190+0000 7fcdb1103640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac0fe6d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcda8006010 tx=0x7fcda80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.191+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcda803f070 con 0x7fcdac102750 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.191+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdac0ff210 con 0x7fcdac102750 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.191+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fcda80049d0 con 0x7fcdac102750 2024-09-21T01:28:00.193 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.191+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdac1a5c80 con 0x7fcdac102750 2024-09-21T01:28:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.191+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fcda8035030 con 0x7fcdac102750 2024-09-21T01:28:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.192+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdac1039d0 con 0x7fcdac102750 2024-09-21T01:28:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.193+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fcda8042070 con 0x7fcdac102750 2024-09-21T01:28:00.194 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.193+0000 7fcda27fc640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcd9807af30 0x7fcd9807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.193+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fcda80bea50 con 0x7fcdac102750 2024-09-21T01:28:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.193+0000 7fcdb0902640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcd9807af30 0x7fcd9807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:00.195 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.194+0000 7fcdb0902640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcd9807af30 0x7fcd9807d440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcdac104690 tx=0x7fcd9c005d30 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:00.199 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.198+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fcda808a250 con 0x7fcdac102750 2024-09-21T01:28:00.200 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.200+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc2b419a4d0 con 0x7fc2b4109190 2024-09-21T01:28:00.201 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.200+0000 7fc2b0ff9640 1 -- 172.21.15.45:0/1764346097 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc2a408ef90 con 0x7fc2b4109190 2024-09-21T01:28:00.201 INFO:teuthology.orchestra.run.smithi045.stdout:115964117019 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc2a007af80 msgr2=0x7fc2a007d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc2a007af80 0x7fc2a007d490 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc29c00abe0 tx=0x7fc29c00a5c0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 msgr2=0x7fc2b4199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b4199df0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc2a402a9c0 tx=0x7fc2a40045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 shutdown_connections 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fc2a007af80 0x7fc2a007d490 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2b4109190 0x7fc2b4199df0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 --2- 172.21.15.45:0/1764346097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc2b4103850 0x7fc2b4199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.202+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 >> 172.21.15.45:0/1764346097 conn(0x7fc2b40ff2a0 msgr2=0x7fc2b41016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.203+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 shutdown_connections 2024-09-21T01:28:00.204 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.203+0000 7fc2b9b16640 1 -- 172.21.15.45:0/1764346097 wait complete. 2024-09-21T01:28:00.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.390+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fcdac1a5e50 con 0x7fcdac102750 2024-09-21T01:28:00.392 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.391+0000 7fcda27fc640 1 -- 172.21.15.45:0/1765492298 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcda808f100 con 0x7fcdac102750 2024-09-21T01:28:00.393 INFO:teuthology.orchestra.run.smithi045.stdout:214748364806 2024-09-21T01:28:00.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcd9807af30 msgr2=0x7fcd9807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcd9807af30 0x7fcd9807d440 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcdac104690 tx=0x7fcd9c005d30 comp rx=0 tx=0).stop 2024-09-21T01:28:00.394 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 msgr2=0x7fcdac0fe6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:00.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac0fe6d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcda8006010 tx=0x7fcda80045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 shutdown_connections 2024-09-21T01:28:00.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fcd9807af30 0x7fcd9807d440 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcdac108090 0x7fcdac0fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 --2- 172.21.15.45:0/1765492298 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcdac102750 0x7fcdac0fe6d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:00.395 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 >> 172.21.15.45:0/1765492298 conn(0x7fcdac0fe1a0 msgr2=0x7fcdac101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:00.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 shutdown_connections 2024-09-21T01:28:00.396 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:00.393+0000 7fcdb3367640 1 -- 172.21.15.45:0/1765492298 wait complete. 2024-09-21T01:28:00.968 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:00 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1764346097' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T01:28:00.968 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:00 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1765492298' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T01:28:00.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:00 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1764346097' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T01:28:00.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:00 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1765492298' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T01:28:01.004 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364805 got 214748364806 for osd.7 2024-09-21T01:28:01.004 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:01.615 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:01.615 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.647+0000 7f1dcd542640 1 -- 172.21.15.45:0/40862779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc8101d50 msgr2=0x7f1dc8102160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.647+0000 7f1dcd542640 1 --2- 172.21.15.45:0/40862779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc8101d50 0x7f1dc8102160 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009880 tx=0x7f1db802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.647+0000 7f1dcd542640 1 -- 172.21.15.45:0/40862779 shutdown_connections 2024-09-21T01:28:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.647+0000 7f1dcd542640 1 --2- 172.21.15.45:0/40862779 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc8105fa0 0x7f1dc81063b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.647+0000 7f1dcd542640 1 --2- 172.21.15.45:0/40862779 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc8101d50 0x7f1dc8102160 secure :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009880 tx=0x7f1db802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.649 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.647+0000 7f1dcd542640 1 -- 172.21.15.45:0/40862779 >> 172.21.15.45:0/40862779 conn(0x7f1dc80779a0 msgr2=0x7f1dc8077de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:01.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.648+0000 7f1dcd542640 1 -- 172.21.15.45:0/40862779 shutdown_connections 2024-09-21T01:28:01.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.648+0000 7f1dcd542640 1 -- 172.21.15.45:0/40862779 wait complete. 2024-09-21T01:28:01.650 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.649+0000 7f1dcd542640 1 Processor -- start 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.649+0000 7f1dcd542640 1 -- start start 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.649+0000 7f1dcd542640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc8105fa0 0x7f1dc8199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dcd542640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 0x7f1dc819f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dcd542640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc8110930 con 0x7f1dc819a010 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dcd542640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f1dc81107b0 con 0x7f1dc8105fa0 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 0x7f1dc819f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc67fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 0x7f1dc819f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58642/0 (socket says 172.21.15.45:58642) 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc67fc640 1 -- 172.21.15.45:0/3251403693 learned_addr learned my addr 172.21.15.45:0/3251403693 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:01.651 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc6ffd640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc8105fa0 0x7f1dc8199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc67fc640 1 -- 172.21.15.45:0/3251403693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc8105fa0 msgr2=0x7f1dc8199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc67fc640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc8105fa0 0x7f1dc8199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.650+0000 7f1dc67fc640 1 -- 172.21.15.45:0/3251403693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dc819f790 con 0x7f1dc819a010 2024-09-21T01:28:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.651+0000 7f1dc67fc640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 0x7f1dc819f250 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00c840 tx=0x7f1dbc00cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:01.652 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.651+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1dbc0079b0 con 0x7f1dc819a010 2024-09-21T01:28:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.651+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dc819fa80 con 0x7f1dc819a010 2024-09-21T01:28:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.651+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f1dbc00f450 con 0x7f1dc819a010 2024-09-21T01:28:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.651+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dc81a0010 con 0x7f1dc819a010 2024-09-21T01:28:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.652+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1dbc005020 con 0x7f1dc819a010 2024-09-21T01:28:01.653 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.652+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dc806a8c0 con 0x7f1dc819a010 2024-09-21T01:28:01.654 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.653+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f1dbc007b50 con 0x7f1dc819a010 2024-09-21T01:28:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.654+0000 7f1daffff640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f1db007af30 0x7f1db007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.654+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f1dbc09de80 con 0x7f1dc819a010 2024-09-21T01:28:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.654+0000 7f1dc6ffd640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f1db007af30 0x7f1db007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:01.655 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.654+0000 7f1dc6ffd640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f1db007af30 0x7f1db007d440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1db8006010 tx=0x7f1db8004220 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:01.660 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.659+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f1dbc014030 con 0x7f1dc819a010 2024-09-21T01:28:01.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.852+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1dc810b980 con 0x7f1dc819a010 2024-09-21T01:28:01.854 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:01 smithi045 ceph-mon[28361]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:01.854 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.853+0000 7f1daffff640 1 -- 172.21.15.45:0/3251403693 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1dbc069680 con 0x7f1dc819a010 2024-09-21T01:28:01.855 INFO:teuthology.orchestra.run.smithi045.stdout:38654705708 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.855+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f1db007af30 msgr2=0x7f1db007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.855+0000 7f1dcd542640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f1db007af30 0x7f1db007d440 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1db8006010 tx=0x7f1db8004220 comp rx=0 tx=0).stop 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.855+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 msgr2=0x7f1dc819f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.855+0000 7f1dcd542640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 0x7f1dc819f250 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc00c840 tx=0x7f1dbc00cd00 comp rx=0 tx=0).stop 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 shutdown_connections 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f1db007af30 0x7f1db007d440 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc819a010 0x7f1dc819f250 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 --2- 172.21.15.45:0/3251403693 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1dc8105fa0 0x7f1dc8199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 >> 172.21.15.45:0/3251403693 conn(0x7f1dc80779a0 msgr2=0x7f1dc8107390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 shutdown_connections 2024-09-21T01:28:01.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.856+0000 7f1dcd542640 1 -- 172.21.15.45:0/3251403693 wait complete. 2024-09-21T01:28:01.971 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 -- 172.21.15.45:0/2135950368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4108090 msgr2=0x7f65f41084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 --2- 172.21.15.45:0/2135950368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4108090 0x7f65f41084a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009880 tx=0x7f65e802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 -- 172.21.15.45:0/2135950368 shutdown_connections 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 --2- 172.21.15.45:0/2135950368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4108090 0x7f65f41084a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 --2- 172.21.15.45:0/2135950368 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f4102750 0x7f65f4102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 -- 172.21.15.45:0/2135950368 >> 172.21.15.45:0/2135950368 conn(0x7f65f40fe1a0 msgr2=0x7f65f41005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 -- 172.21.15.45:0/2135950368 shutdown_connections 2024-09-21T01:28:01.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.970+0000 7f65f952c640 1 -- 172.21.15.45:0/2135950368 wait complete. 2024-09-21T01:28:01.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.971+0000 7f65f952c640 1 Processor -- start 2024-09-21T01:28:01.973 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f952c640 1 -- start start 2024-09-21T01:28:01.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f952c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 0x7f65f419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:01.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f952c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f4108090 0x7f65f419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:01.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f952c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f65f410f840 con 0x7f65f4102750 2024-09-21T01:28:01.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f952c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f65f410f6c0 con 0x7f65f4108090 2024-09-21T01:28:01.974 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 0x7f65f419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 0x7f65f419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58666/0 (socket says 172.21.15.45:58666) 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f2ffd640 1 -- 172.21.15.45:0/4184214270 learned_addr learned my addr 172.21.15.45:0/4184214270 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.972+0000 7f65f27fc640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f4108090 0x7f65f419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65f2ffd640 1 -- 172.21.15.45:0/4184214270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f4108090 msgr2=0x7f65f419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65f2ffd640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f4108090 0x7f65f419acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65f2ffd640 1 -- 172.21.15.45:0/4184214270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f419b210 con 0x7f65f4102750 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65f2ffd640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 0x7f65f419a790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f65e400b420 tx=0x7f65e400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f65e4011630 con 0x7f65f4102750 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f65e4016070 con 0x7f65f4102750 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f419b500 con 0x7f65f4102750 2024-09-21T01:28:01.975 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.973+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f65e4010b80 con 0x7f65f4102750 2024-09-21T01:28:01.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.974+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f41a3aa0 con 0x7f65f4102750 2024-09-21T01:28:01.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.975+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f41039d0 con 0x7f65f4102750 2024-09-21T01:28:01.976 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.975+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f65e401b460 con 0x7f65f4102750 2024-09-21T01:28:01.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.976+0000 7f65dbfff640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f65dc07af80 0x7f65dc07d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:01.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.976+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f65e409d4d0 con 0x7f65f4102750 2024-09-21T01:28:01.977 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.976+0000 7f65f27fc640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f65dc07af80 0x7f65dc07d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:01.978 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.977+0000 7f65f27fc640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f65dc07af80 0x7f65dc07d490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009230 tx=0x7f65e80058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:01.981 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:01.980+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f65e4068d50 con 0x7f65f4102750 2024-09-21T01:28:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:01 smithi090 ceph-mon[35142]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:02.179 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.177+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f65f410a890 con 0x7f65f4102750 2024-09-21T01:28:02.180 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.178+0000 7f65dbfff640 1 -- 172.21.15.45:0/4184214270 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f65e406dc00 con 0x7f65f4102750 2024-09-21T01:28:02.180 INFO:teuthology.orchestra.run.smithi045.stdout:60129542183 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f65dc07af80 msgr2=0x7f65dc07d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f65dc07af80 0x7f65dc07d490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009230 tx=0x7f65e80058d0 comp rx=0 tx=0).stop 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 msgr2=0x7f65f419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 0x7f65f419a790 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f65e400b420 tx=0x7f65e400b8e0 comp rx=0 tx=0).stop 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 shutdown_connections 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f65dc07af80 0x7f65dc07d490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:02.182 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f65f4108090 0x7f65f419acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 --2- 172.21.15.45:0/4184214270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f65f4102750 0x7f65f419a790 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:02.183 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.180+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 >> 172.21.15.45:0/4184214270 conn(0x7f65f40fe1a0 msgr2=0x7f65f41005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:02.185 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.181+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 shutdown_connections 2024-09-21T01:28:02.186 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:02.181+0000 7f65f952c640 1 -- 172.21.15.45:0/4184214270 wait complete. 2024-09-21T01:28:02.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:02 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3251403693' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T01:28:02.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:02 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4184214270' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T01:28:02.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:02 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3251403693' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T01:28:02.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:02 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4184214270' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T01:28:03.043 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117018 got 115964117019 for osd.3 2024-09-21T01:28:03.044 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:03.306 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:03.610 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705707 got 38654705708 for osd.0 2024-09-21T01:28:03.610 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:03.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/1828489789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 msgr2=0x7f6b78102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:03.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/1828489789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78102910 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009880 tx=0x7f6b6c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/1828489789 shutdown_connections 2024-09-21T01:28:03.751 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/1828489789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b78107e40 0x7f6b78108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/1828489789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78102910 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/1828489789 >> 172.21.15.45:0/1828489789 conn(0x7f6b780fdf30 msgr2=0x7f6b78100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:03.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/1828489789 shutdown_connections 2024-09-21T01:28:03.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.750+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/1828489789 wait complete. 2024-09-21T01:28:03.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7dd9d640 1 Processor -- start 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7dd9d640 1 -- start start 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7dd9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7dd9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b78107e40 0x7f6b78198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7dd9d640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7810f5d0 con 0x7f6b78102500 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7dd9d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7810f450 con 0x7f6b78107e40 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7cd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7cd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58684/0 (socket says 172.21.15.45:58684) 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.751+0000 7f6b7cd9b640 1 -- 172.21.15.45:0/4091311878 learned_addr learned my addr 172.21.15.45:0/4091311878 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b77fff640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b78107e40 0x7f6b78198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b7cd9b640 1 -- 172.21.15.45:0/4091311878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b78107e40 msgr2=0x7f6b78198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b7cd9b640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b78107e40 0x7f6b78198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b7cd9b640 1 -- 172.21.15.45:0/4091311878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7819dd60 con 0x7f6b78102500 2024-09-21T01:28:03.754 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b7cd9b640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78198520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c02a9c0 tx=0x7f6b6c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6b6c03f070 con 0x7f6b78102500 2024-09-21T01:28:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7819dff0 con 0x7f6b78102500 2024-09-21T01:28:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6b6c004a90 con 0x7f6b78102500 2024-09-21T01:28:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.752+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6b6c035030 con 0x7f6b78102500 2024-09-21T01:28:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.753+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7819e5a0 con 0x7f6b78102500 2024-09-21T01:28:03.755 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.754+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b78103780 con 0x7f6b78102500 2024-09-21T01:28:03.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.755+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f6b6c042070 con 0x7f6b78102500 2024-09-21T01:28:03.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.755+0000 7f6b75ffb640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6b6807af30 0x7f6b6807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:03.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.755+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f6b6c0be800 con 0x7f6b78102500 2024-09-21T01:28:03.756 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.755+0000 7f6b77fff640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6b6807af30 0x7f6b6807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:03.757 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.756+0000 7f6b77fff640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6b6807af30 0x7f6b6807d440 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6b78107930 tx=0x7f6b6400b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:03.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.760+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f6b6c08a000 con 0x7f6b78102500 2024-09-21T01:28:03.846 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:03 smithi045 ceph-mon[28361]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:03.952 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.951+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f6b7810a5e0 con 0x7f6b78102500 2024-09-21T01:28:03.953 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.952+0000 7f6b75ffb640 1 -- 172.21.15.45:0/4091311878 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6b6c08eeb0 con 0x7f6b78102500 2024-09-21T01:28:03.953 INFO:teuthology.orchestra.run.smithi045.stdout:163208757266 2024-09-21T01:28:03.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6b6807af30 msgr2=0x7f6b6807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:03.955 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6b6807af30 0x7f6b6807d440 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6b78107930 tx=0x7f6b6400b390 comp rx=0 tx=0).stop 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 msgr2=0x7f6b78198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78198520 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c02a9c0 tx=0x7f6b6c0045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 shutdown_connections 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6b6807af30 0x7f6b6807d440 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6b78107e40 0x7f6b78198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 --2- 172.21.15.45:0/4091311878 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6b78102500 0x7f6b78198520 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 >> 172.21.15.45:0/4091311878 conn(0x7f6b780fdf30 msgr2=0x7f6b78100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 shutdown_connections 2024-09-21T01:28:03.956 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:03.954+0000 7f6b7dd9d640 1 -- 172.21.15.45:0/4091311878 wait complete. 2024-09-21T01:28:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:03 smithi090 ceph-mon[35142]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:04.175 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542182 got 60129542183 for osd.1 2024-09-21T01:28:04.176 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:04.824 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757264 got 163208757266 for osd.5 2024-09-21T01:28:04.825 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:04.910 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:04 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4091311878' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T01:28:04.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:04 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4091311878' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T01:28:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 -- 172.21.15.45:0/3652855234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec1091a0 msgr2=0x7f58ec1095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 --2- 172.21.15.45:0/3652855234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec1091a0 0x7f58ec1095b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009880 tx=0x7f58dc02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 -- 172.21.15.45:0/3652855234 shutdown_connections 2024-09-21T01:28:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 --2- 172.21.15.45:0/3652855234 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec1091a0 0x7f58ec1095b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.008 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 --2- 172.21.15.45:0/3652855234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec103860 0x7f58ec103c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 -- 172.21.15.45:0/3652855234 >> 172.21.15.45:0/3652855234 conn(0x7f58ec0ff2f0 msgr2=0x7f58ec101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:05.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 -- 172.21.15.45:0/3652855234 shutdown_connections 2024-09-21T01:28:05.009 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.007+0000 7f58f3e57640 1 -- 172.21.15.45:0/3652855234 wait complete. 2024-09-21T01:28:05.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f3e57640 1 Processor -- start 2024-09-21T01:28:05.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f3e57640 1 -- start start 2024-09-21T01:28:05.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f3e57640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 0x7f58ec075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:05.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f3e57640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec1091a0 0x7f58ec075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:05.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f3e57640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec1108f0 con 0x7f58ec103860 2024-09-21T01:28:05.010 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f3e57640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f58ec110770 con 0x7f58ec1091a0 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f1bf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 0x7f58ec075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f1bf3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 0x7f58ec075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58694/0 (socket says 172.21.15.45:58694) 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f1bf3640 1 -- 172.21.15.45:0/2090663205 learned_addr learned my addr 172.21.15.45:0/2090663205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.008+0000 7f58f13f2640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec1091a0 0x7f58ec075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58f1bf3640 1 -- 172.21.15.45:0/2090663205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec1091a0 msgr2=0x7f58ec075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58f1bf3640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec1091a0 0x7f58ec075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58f1bf3640 1 -- 172.21.15.45:0/2090663205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58ec0775f0 con 0x7f58ec103860 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58f1bf3640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 0x7f58ec075710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c8f0 tx=0x7f58e800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f58e80079b0 con 0x7f58ec103860 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f58e8005020 con 0x7f58ec103860 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58ec0762b0 con 0x7f58ec103860 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f58e8004c50 con 0x7f58ec103860 2024-09-21T01:28:05.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.009+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58ec1a5bc0 con 0x7f58ec103860 2024-09-21T01:28:05.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.010+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58ec104ae0 con 0x7f58ec103860 2024-09-21T01:28:05.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.011+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f58e80051d0 con 0x7f58ec103860 2024-09-21T01:28:05.012 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.011+0000 7f58e2ffd640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f58d807af40 0x7f58d807d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:05.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.011+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f58e809dc70 con 0x7f58ec103860 2024-09-21T01:28:05.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.011+0000 7f58f13f2640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f58d807af40 0x7f58d807d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:05.013 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.012+0000 7f58f13f2640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f58d807af40 0x7f58d807d450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009230 tx=0x7f58dc005a20 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:05.017 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.016+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f58e8014030 con 0x7f58ec103860 2024-09-21T01:28:05.039 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.211+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f58ec103c70 con 0x7f58ec103860 2024-09-21T01:28:05.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.211+0000 7f58e2ffd640 1 -- 172.21.15.45:0/2090663205 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f58e8069470 con 0x7f58ec103860 2024-09-21T01:28:05.212 INFO:teuthology.orchestra.run.smithi045.stdout:137438953495 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.213+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f58d807af40 msgr2=0x7f58d807d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f58d807af40 0x7f58d807d450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009230 tx=0x7f58dc005a20 comp rx=0 tx=0).stop 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 msgr2=0x7f58ec075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 0x7f58ec075710 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f58e800c8f0 tx=0x7f58e800cdb0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 shutdown_connections 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f58d807af40 0x7f58d807d450 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f58ec1091a0 0x7f58ec075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 --2- 172.21.15.45:0/2090663205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f58ec103860 0x7f58ec075710 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:05.215 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 >> 172.21.15.45:0/2090663205 conn(0x7f58ec0ff2f0 msgr2=0x7f58ec101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 shutdown_connections 2024-09-21T01:28:05.216 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:05.214+0000 7f58f3e57640 1 -- 172.21.15.45:0/2090663205 wait complete. 2024-09-21T01:28:05.900 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:05 smithi045 ceph-mon[28361]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:05.900 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:05 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2090663205' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T01:28:05.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953493 got 137438953495 for osd.4 2024-09-21T01:28:05.931 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:05 smithi090 ceph-mon[35142]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:05 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2090663205' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.363+0000 7f47c3041640 1 -- 172.21.15.45:0/2752866560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc1046d0 msgr2=0x7f47bc104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.363+0000 7f47c3041640 1 --2- 172.21.15.45:0/2752866560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc1046d0 0x7f47bc104ae0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009880 tx=0x7f47b002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 -- 172.21.15.45:0/2752866560 shutdown_connections 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 --2- 172.21.15.45:0/2752866560 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47bc105020 0x7f47bc10d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 --2- 172.21.15.45:0/2752866560 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc1046d0 0x7f47bc104ae0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 -- 172.21.15.45:0/2752866560 >> 172.21.15.45:0/2752866560 conn(0x7f47bc0fe1d0 msgr2=0x7f47bc100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:06.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 -- 172.21.15.45:0/2752866560 shutdown_connections 2024-09-21T01:28:06.366 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 -- 172.21.15.45:0/2752866560 wait complete. 2024-09-21T01:28:06.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.364+0000 7f47c3041640 1 Processor -- start 2024-09-21T01:28:06.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47c3041640 1 -- start start 2024-09-21T01:28:06.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47c3041640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47bc1046d0 0x7f47bc102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:06.367 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47c3041640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 0x7f47bc102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47c3041640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc110940 con 0x7f47bc105020 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47c3041640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc1107c0 con 0x7f47bc1046d0 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 0x7f47bc102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 0x7f47bc102950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58718/0 (socket says 172.21.15.45:58718) 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 -- 172.21.15.45:0/4212222400 learned_addr learned my addr 172.21.15.45:0/4212222400 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47c0ddd640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47bc1046d0 0x7f47bc102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 -- 172.21.15.45:0/4212222400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47bc1046d0 msgr2=0x7f47bc102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47bc1046d0 0x7f47bc102410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 -- 172.21.15.45:0/4212222400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47bc1042f0 con 0x7f47bc105020 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.365+0000 7f47bbfff640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 0x7f47bc102950 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f47a800d6d0 tx=0x7f47a800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.366+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f47a8017070 con 0x7f47bc105020 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.366+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f47a80054b0 con 0x7f47bc105020 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.366+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47bc102fb0 con 0x7f47bc105020 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.366+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f47a80049d0 con 0x7f47bc105020 2024-09-21T01:28:06.368 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.366+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47bc1a5c90 con 0x7f47bc105020 2024-09-21T01:28:06.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.367+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47bc06be10 con 0x7f47bc105020 2024-09-21T01:28:06.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.368+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f47a8009940 con 0x7f47bc105020 2024-09-21T01:28:06.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.368+0000 7f47b9ffb640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f47ac07afc0 0x7f47ac07d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:06.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.368+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f47a809da10 con 0x7f47bc105020 2024-09-21T01:28:06.369 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.368+0000 7f47c0ddd640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f47ac07afc0 0x7f47ac07d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:06.370 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.369+0000 7f47c0ddd640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f47ac07afc0 0x7f47ac07d4d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009230 tx=0x7f47b00058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:06.374 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.373+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f47a8069290 con 0x7f47bc105020 2024-09-21T01:28:06.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.562+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f47bc107340 con 0x7f47bc105020 2024-09-21T01:28:06.563 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.562+0000 7f47b9ffb640 1 -- 172.21.15.45:0/4212222400 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f47a806e140 con 0x7f47bc105020 2024-09-21T01:28:06.564 INFO:teuthology.orchestra.run.smithi045.stdout:188978561037 2024-09-21T01:28:06.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 -- 172.21.15.45:0/3116377188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 msgr2=0x7fe514103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/3116377188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514103c80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe508009880 tx=0x7fe50802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 -- 172.21.15.45:0/3116377188 shutdown_connections 2024-09-21T01:28:06.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/3116377188 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 0x7fe5141095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/3116377188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514103c80 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 -- 172.21.15.45:0/3116377188 >> 172.21.15.45:0/3116377188 conn(0x7fe5140ff300 msgr2=0x7fe514101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:06.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 -- 172.21.15.45:0/3116377188 shutdown_connections 2024-09-21T01:28:06.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.564+0000 7fe51b6fa640 1 -- 172.21.15.45:0/3116377188 wait complete. 2024-09-21T01:28:06.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f47ac07afc0 msgr2=0x7f47ac07d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f47ac07afc0 0x7f47ac07d4d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f47b0009230 tx=0x7f47b00058d0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 msgr2=0x7f47bc102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 0x7f47bc102950 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f47a800d6d0 tx=0x7f47a800db90 comp rx=0 tx=0).stop 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 shutdown_connections 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f47ac07afc0 0x7f47ac07d4d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47bc105020 0x7f47bc102950 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 --2- 172.21.15.45:0/4212222400 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f47bc1046d0 0x7f47bc102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 >> 172.21.15.45:0/4212222400 conn(0x7f47bc0fe1d0 msgr2=0x7f47bc10bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 shutdown_connections 2024-09-21T01:28:06.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7f47c3041640 1 -- 172.21.15.45:0/4212222400 wait complete. 2024-09-21T01:28:06.569 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7fe51b6fa640 1 Processor -- start 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7fe51b6fa640 1 -- start start 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7fe51b6fa640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7fe51b6fa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 0x7fe514075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.565+0000 7fe51b6fa640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe514110920 con 0x7fe514103870 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe51b6fa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5141107a0 con 0x7fe5141091b0 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48880/0 (socket says 172.21.15.45:48880) 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 -- 172.21.15.45:0/1260063090 learned_addr learned my addr 172.21.15.45:0/1260063090 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe518c95640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 0x7fe514075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 -- 172.21.15.45:0/1260063090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 msgr2=0x7fe514075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 0x7fe514075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 -- 172.21.15.45:0/1260063090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5140775f0 con 0x7fe514103870 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe518c95640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 0x7fe514075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe519496640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514075710 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe50802a9c0 tx=0x7fe5080045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe50803f070 con 0x7fe514103870 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.566+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe514076250 con 0x7fe514103870 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.567+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe508004a90 con 0x7fe514103870 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.567+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5141a5c10 con 0x7fe514103870 2024-09-21T01:28:06.572 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.567+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe508035030 con 0x7fe514103870 2024-09-21T01:28:06.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.568+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe514104af0 con 0x7fe514103870 2024-09-21T01:28:06.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.569+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fe508042070 con 0x7fe514103870 2024-09-21T01:28:06.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.569+0000 7fe5067fc640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe50007af30 0x7fe50007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:06.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.569+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fe5080be800 con 0x7fe514103870 2024-09-21T01:28:06.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.569+0000 7fe518c95640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe50007af30 0x7fe50007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:06.573 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.570+0000 7fe518c95640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe50007af30 0x7fe50007d440 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe514105880 tx=0x7fe51000b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:06.575 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.574+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe50808a000 con 0x7fe514103870 2024-09-21T01:28:06.768 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.767+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe51410b950 con 0x7fe514103870 2024-09-21T01:28:06.769 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:06 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4212222400' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T01:28:06.769 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.767+0000 7fe5067fc640 1 -- 172.21.15.45:0/1260063090 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe50808eeb0 con 0x7fe514103870 2024-09-21T01:28:06.769 INFO:teuthology.orchestra.run.smithi045.stdout:81604378659 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe50007af30 msgr2=0x7fe50007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe50007af30 0x7fe50007d440 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe514105880 tx=0x7fe51000b390 comp rx=0 tx=0).stop 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 msgr2=0x7fe514075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514075710 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe50802a9c0 tx=0x7fe5080045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 shutdown_connections 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe50007af30 0x7fe50007d440 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.771 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5141091b0 0x7fe514075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 --2- 172.21.15.45:0/1260063090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe514103870 0x7fe514075710 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:06.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 >> 172.21.15.45:0/1260063090 conn(0x7fe5140ff300 msgr2=0x7fe514101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:06.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 shutdown_connections 2024-09-21T01:28:06.772 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:06.770+0000 7fe51b6fa640 1 -- 172.21.15.45:0/1260063090 wait complete. 2024-09-21T01:28:06.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:06 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4212222400' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T01:28:07.428 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561036 got 188978561037 for osd.6 2024-09-21T01:28:07.428 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:07.830 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378656 got 81604378659 for osd.2 2024-09-21T01:28:07.830 DEBUG:teuthology.parallel:result is None 2024-09-21T01:28:07.831 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T01:28:07.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:28:07.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:07 smithi090 ceph-mon[35142]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:07.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:07 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1260063090' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T01:28:07.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:08.093 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:07 smithi045 ceph-mon[28361]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:07 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1260063090' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T01:28:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 -- 172.21.15.45:0/403467393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 msgr2=0x7feae41084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 --2- 172.21.15.45:0/403467393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae41084a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fead8009880 tx=0x7fead802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 -- 172.21.15.45:0/403467393 shutdown_connections 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 --2- 172.21.15.45:0/403467393 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae41084a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 --2- 172.21.15.45:0/403467393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae4102750 0x7feae4102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 -- 172.21.15.45:0/403467393 >> 172.21.15.45:0/403467393 conn(0x7feae40fe1a0 msgr2=0x7feae41005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 -- 172.21.15.45:0/403467393 shutdown_connections 2024-09-21T01:28:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.538+0000 7feae9420640 1 -- 172.21.15.45:0/403467393 wait complete. 2024-09-21T01:28:08.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae9420640 1 Processor -- start 2024-09-21T01:28:08.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae9420640 1 -- start start 2024-09-21T01:28:08.541 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae9420640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae4102750 0x7feae41998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae9420640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae4199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae9420640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7feae410f7e0 con 0x7feae4108090 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae9420640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7feae410f660 con 0x7feae4102750 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.539+0000 7feae27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae4199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae27fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae4199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48896/0 (socket says 172.21.15.45:48896) 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae27fc640 1 -- 172.21.15.45:0/1465673048 learned_addr learned my addr 172.21.15.45:0/1465673048 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae2ffd640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae4102750 0x7feae41998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae27fc640 1 -- 172.21.15.45:0/1465673048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae4102750 msgr2=0x7feae41998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae27fc640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae4102750 0x7feae41998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae27fc640 1 -- 172.21.15.45:0/1465673048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae419f100 con 0x7feae4108090 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae27fc640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae4199e60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fead802a9c0 tx=0x7fead80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fead803f070 con 0x7feae4108090 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae419f390 con 0x7feae4108090 2024-09-21T01:28:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fead8004950 con 0x7feae4108090 2024-09-21T01:28:08.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fead8035030 con 0x7feae4108090 2024-09-21T01:28:08.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.540+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae419f8c0 con 0x7feae4108090 2024-09-21T01:28:08.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.541+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae41039d0 con 0x7feae4108090 2024-09-21T01:28:08.543 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.542+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fead8042070 con 0x7feae4108090 2024-09-21T01:28:08.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.543+0000 7feacbfff640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feacc07af30 0x7feacc07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:08.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.543+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fead80be800 con 0x7feae4108090 2024-09-21T01:28:08.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.543+0000 7feae2ffd640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feacc07af30 0x7feacc07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:08.544 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.543+0000 7feae2ffd640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feacc07af30 0x7feacc07d440 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fead4005fd0 tx=0x7fead400b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:08.548 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.547+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fead808a000 con 0x7feae4108090 2024-09-21T01:28:08.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.731+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feae419a540 con 0x7feacc07af30 2024-09-21T01:28:08.733 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.732+0000 7feacbfff640 1 -- 172.21.15.45:0/1465673048 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+14509 (secure 0 0 0) 0x7feae419a540 con 0x7feacc07af30 2024-09-21T01:28:08.734 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:08.734 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-21T01:28:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feacc07af30 msgr2=0x7feacc07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feacc07af30 0x7feacc07d440 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fead4005fd0 tx=0x7fead400b390 comp rx=0 tx=0).stop 2024-09-21T01:28:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 msgr2=0x7feae4199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:08.736 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae4199e60 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fead802a9c0 tx=0x7fead80045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 shutdown_connections 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7feacc07af30 0x7feacc07d440 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7feae4108090 0x7feae4199e60 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 --2- 172.21.15.45:0/1465673048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feae4102750 0x7feae41998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.734+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 >> 172.21.15.45:0/1465673048 conn(0x7feae40fe1a0 msgr2=0x7feae4101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.735+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 shutdown_connections 2024-09-21T01:28:08.737 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:08.735+0000 7feae9420640 1 -- 172.21.15.45:0/1465673048 wait complete. 2024-09-21T01:28:09.462 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-21T01:28:08.011810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":222072,"kb_used_data":5804,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749739144,"statfs":{"total":767960285184,"available":767732883456,"internally_reserved":0,"allocated":5943296,"data_stored":3437496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002525"},"pg_stats":[{"pgid":"1.0","version":"22'72","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-21T01:27:38.634889+0000","last_change":"2024-09-21T01:27:38.632705+0000","last_active":"2024-09-21T01:27:38.634889+0000","last_peered":"2024-09-21T01:27:38.634889+0000","last_clean":"2024-09-21T01:27:38.634889+0000","last_became_active":"2024-09-21T01:27:38.624695+0000","last_became_peered":"2024-09-21T01:27:38.624695+0000","last_unstale":"2024-09-21T01:27:38.634889+0000","last_undegraded":"2024-09-21T01:27:38.634889+0000","last_fullsized":"2024-09-21T01:27:38.634889+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:25:31.049693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:25:31.049693+0000","last_clean_scrub_stamp":"2024-09-21T01:25:31.049693+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:06.803162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1114112,"data_stored":798667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1114112,"data_stored":798667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27540,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717612,"statfs":{"total":95995035648,"available":95966834688,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953496,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27536,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717616,"statfs":{"total":95995035648,"available":95966838784,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717608,"statfs":{"total":95995035648,"available":95966830592,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717608,"statfs":{"total":95995035648,"available":95966830592,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717608,"statfs":{"total":95995035648,"available":95966830592,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1114112,"data_stored":798667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:28:09.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-21T01:28:09.721 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:09.842 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:09 smithi045 ceph-mon[28361]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:09.842 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:09 smithi045 ceph-mon[28361]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:28:09.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:09 smithi090 ceph-mon[35142]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:09.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:09 smithi090 ceph-mon[35142]: from='client.14580 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:28:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 -- 172.21.15.45:0/1170636685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 msgr2=0x7efe181084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 --2- 172.21.15.45:0/1170636685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe181084a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efe0c009880 tx=0x7efe0c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 -- 172.21.15.45:0/1170636685 shutdown_connections 2024-09-21T01:28:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 --2- 172.21.15.45:0/1170636685 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe181084a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 --2- 172.21.15.45:0/1170636685 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 0x7efe18102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.164 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 -- 172.21.15.45:0/1170636685 >> 172.21.15.45:0/1170636685 conn(0x7efe180fe1a0 msgr2=0x7efe181005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 -- 172.21.15.45:0/1170636685 shutdown_connections 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.161+0000 7efe1d91e640 1 -- 172.21.15.45:0/1170636685 wait complete. 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe1d91e640 1 Processor -- start 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe1d91e640 1 -- start start 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe1d91e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 0x7efe181999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe1d91e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe18199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe1d91e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810f840 con 0x7efe18108090 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe1d91e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7efe1810f6c0 con 0x7efe18102750 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe167fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe18199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe167fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe18199f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48914/0 (socket says 172.21.15.45:48914) 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.162+0000 7efe167fc640 1 -- 172.21.15.45:0/3789715964 learned_addr learned my addr 172.21.15.45:0/3789715964 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe16ffd640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 0x7efe181999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe167fc640 1 -- 172.21.15.45:0/3789715964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 msgr2=0x7efe181999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:10.165 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe167fc640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 0x7efe181999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe167fc640 1 -- 172.21.15.45:0/3789715964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1819f210 con 0x7efe18108090 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe16ffd640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 0x7efe181999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe167fc640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe18199f70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe0c02a9c0 tx=0x7efe0c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7efe0c03f070 con 0x7efe18108090 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7efe0c0049d0 con 0x7efe18108090 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1819f4a0 con 0x7efe18108090 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7efe0c035030 con 0x7efe18108090 2024-09-21T01:28:10.166 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.163+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1819f9d0 con 0x7efe18108090 2024-09-21T01:28:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.164+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe181039d0 con 0x7efe18108090 2024-09-21T01:28:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.165+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7efe0c042070 con 0x7efe18108090 2024-09-21T01:28:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.166+0000 7efe1c91c640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7efe0007af30 0x7efe0007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.166+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7efe0c0be9d0 con 0x7efe18108090 2024-09-21T01:28:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.166+0000 7efe16ffd640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7efe0007af30 0x7efe0007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:10.167 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.166+0000 7efe16ffd640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7efe0007af30 0x7efe0007d440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe08006fd0 tx=0x7efe08008040 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:10.171 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.170+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7efe0c08a130 con 0x7efe18108090 2024-09-21T01:28:10.358 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.357+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efe1819a650 con 0x7efe0007af30 2024-09-21T01:28:10.359 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.358+0000 7efe1c91c640 1 -- 172.21.15.45:0/3789715964 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+14509 (secure 0 0 0) 0x7efe1819a650 con 0x7efe0007af30 2024-09-21T01:28:10.359 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:10.359 INFO:teuthology.orchestra.run.smithi045.stderr:dumped all 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7efe0007af30 msgr2=0x7efe0007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7efe0007af30 0x7efe0007d440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efe08006fd0 tx=0x7efe08008040 comp rx=0 tx=0).stop 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 msgr2=0x7efe18199f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe18199f70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe0c02a9c0 tx=0x7efe0c0045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 shutdown_connections 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7efe0007af30 0x7efe0007d440 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe18108090 0x7efe18199f70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 --2- 172.21.15.45:0/3789715964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efe18102750 0x7efe181999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:10.361 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 >> 172.21.15.45:0/3789715964 conn(0x7efe180fe1a0 msgr2=0x7efe181011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 shutdown_connections 2024-09-21T01:28:10.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:10.360+0000 7efe1d91e640 1 -- 172.21.15.45:0/3789715964 wait complete. 2024-09-21T01:28:11.027 INFO:teuthology.orchestra.run.smithi045.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-21T01:28:10.020896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":222072,"kb_used_data":5804,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749739144,"statfs":{"total":767960285184,"available":767732883456,"internally_reserved":0,"allocated":5943296,"data_stored":3437496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011259"},"pg_stats":[{"pgid":"1.0","version":"22'72","reported_seq":18,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-09-21T01:27:38.634889+0000","last_change":"2024-09-21T01:27:38.632705+0000","last_active":"2024-09-21T01:27:38.634889+0000","last_peered":"2024-09-21T01:27:38.634889+0000","last_clean":"2024-09-21T01:27:38.634889+0000","last_became_active":"2024-09-21T01:27:38.624695+0000","last_became_peered":"2024-09-21T01:27:38.624695+0000","last_unstale":"2024-09-21T01:27:38.634889+0000","last_undegraded":"2024-09-21T01:27:38.634889+0000","last_fullsized":"2024-09-21T01:27:38.634889+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T01:25:31.049693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T01:25:31.049693+0000","last_clean_scrub_stamp":"2024-09-21T01:25:31.049693+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:06.803162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":50,"seq":214748364808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1114112,"data_stored":798667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":44,"seq":188978561038,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28120,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717032,"statfs":{"total":95995035648,"available":95966240768,"internally_reserved":0,"allocated":1114112,"data_stored":798667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757267,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27540,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717612,"statfs":{"total":95995035648,"available":95966834688,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953497,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27536,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717616,"statfs":{"total":95995035648,"available":95966838784,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117021,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717608,"statfs":{"total":95995035648,"available":95966830592,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":19,"seq":81604378659,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717608,"statfs":{"total":95995035648,"available":95966830592,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27544,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717608,"statfs":{"total":95995035648,"available":95966830592,"internally_reserved":0,"allocated":520192,"data_stored":208299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705710,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1114112,"data_stored":798667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T01:28:11.027 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T01:28:11.027 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T01:28:11.027 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T01:28:11.028 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-21T01:28:11.288 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 -- 172.21.15.45:0/2072449558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 msgr2=0x7fd77c104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 --2- 172.21.15.45:0/2072449558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c104ab0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd76c009880 tx=0x7fd76c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 -- 172.21.15.45:0/2072449558 shutdown_connections 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 --2- 172.21.15.45:0/2072449558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd77c104ff0 0x7fd77c10d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 --2- 172.21.15.45:0/2072449558 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c104ab0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 -- 172.21.15.45:0/2072449558 >> 172.21.15.45:0/2072449558 conn(0x7fd77c0fe1a0 msgr2=0x7fd77c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 -- 172.21.15.45:0/2072449558 shutdown_connections 2024-09-21T01:28:11.759 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 -- 172.21.15.45:0/2072449558 wait complete. 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.750+0000 7fd7820b9640 1 Processor -- start 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd7820b9640 1 -- start start 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd7820b9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd7820b9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd77c104ff0 0x7fd77c199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd7820b9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c110910 con 0x7fd77c1046a0 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd7820b9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fd77c110790 con 0x7fd77c104ff0 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77b7fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48934/0 (socket says 172.21.15.45:48934) 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77b7fe640 1 -- 172.21.15.45:0/1586406814 learned_addr learned my addr 172.21.15.45:0/1586406814 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77affd640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd77c104ff0 0x7fd77c199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77b7fe640 1 -- 172.21.15.45:0/1586406814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd77c104ff0 msgr2=0x7fd77c199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77b7fe640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd77c104ff0 0x7fd77c199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.760 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.751+0000 7fd77b7fe640 1 -- 172.21.15.45:0/1586406814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd77c19f080 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.752+0000 7fd77b7fe640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c199840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd76c02a9c0 tx=0x7fd76c004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.752+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd76c03f070 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.752+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fd76c0051c0 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.752+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd77c19f310 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.752+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fd76c035030 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.752+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd77c19f8c0 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.753+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd77c06bdf0 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.754+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fd76c042070 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.754+0000 7fd778ff9640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fd76407af60 0x7fd76407d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.754+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fd76c0be7b0 con 0x7fd77c1046a0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.754+0000 7fd77affd640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fd76407af60 0x7fd76407d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:11.761 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.755+0000 7fd77affd640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fd76407af60 0x7fd76407d470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd77c100680 tx=0x7fd77000b390 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:11.762 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.759+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fd76c089fb0 con 0x7fd77c1046a0 2024-09-21T01:28:11.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:11 smithi045 ceph-mon[28361]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:11 smithi045 ceph-mon[28361]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:28:11.987 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.986+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd77c19a4f0 con 0x7fd77c1046a0 2024-09-21T01:28:11.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.986+0000 7fd778ff9640 1 -- 172.21.15.45:0/1586406814 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd76c08ee60 con 0x7fd77c1046a0 2024-09-21T01:28:11.988 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:11.988 INFO:teuthology.orchestra.run.smithi045.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fd76407af60 msgr2=0x7fd76407d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fd76407af60 0x7fd76407d470 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd77c100680 tx=0x7fd77000b390 comp rx=0 tx=0).stop 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 msgr2=0x7fd77c199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c199840 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd76c02a9c0 tx=0x7fd76c004c60 comp rx=0 tx=0).stop 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 shutdown_connections 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fd76407af60 0x7fd76407d470 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd77c104ff0 0x7fd77c199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 --2- 172.21.15.45:0/1586406814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd77c1046a0 0x7fd77c199840 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 >> 172.21.15.45:0/1586406814 conn(0x7fd77c0fe1a0 msgr2=0x7fd77c10ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 shutdown_connections 2024-09-21T01:28:11.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:11.989+0000 7fd7820b9640 1 -- 172.21.15.45:0/1586406814 wait complete. 2024-09-21T01:28:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:11 smithi090 ceph-mon[35142]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:11 smithi090 ceph-mon[35142]: from='client.14584 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T01:28:12.700 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T01:28:12.700 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T01:28:12.700 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T01:28:12.710 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2024-09-21T01:28:12.710 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-21T01:28:12.970 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:12.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:12 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1586406814' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T01:28:13.091 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:12 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1586406814' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 -- 172.21.15.45:0/909498270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 msgr2=0x7f46d81027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/909498270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d81027c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f46c8009880 tx=0x7f46c802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 -- 172.21.15.45:0/909498270 shutdown_connections 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/909498270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d8103480 0x7f46d8103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/909498270 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d81027c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 -- 172.21.15.45:0/909498270 >> 172.21.15.45:0/909498270 conn(0x7f46d80fdce0 msgr2=0x7f46d8100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 -- 172.21.15.45:0/909498270 shutdown_connections 2024-09-21T01:28:13.427 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.424+0000 7f46dd0f9640 1 -- 172.21.15.45:0/909498270 wait complete. 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46dd0f9640 1 Processor -- start 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46dd0f9640 1 -- start start 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46dd0f9640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d8199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46dd0f9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d8103480 0x7f46d8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46dd0f9640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f46d810b0f0 con 0x7f46d8102350 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46dd0f9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f46d810af70 con 0x7f46d8103480 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46d6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d8199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.425+0000 7f46d6d9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d8199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48958/0 (socket says 172.21.15.45:48958) 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46d6d9d640 1 -- 172.21.15.45:0/3781076208 learned_addr learned my addr 172.21.15.45:0/3781076208 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46d659c640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d8103480 0x7f46d8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46d6d9d640 1 -- 172.21.15.45:0/3781076208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d8103480 msgr2=0x7f46d8199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46d6d9d640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d8103480 0x7f46d8199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46d6d9d640 1 -- 172.21.15.45:0/3781076208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d819a5d0 con 0x7f46d8102350 2024-09-21T01:28:13.428 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46d6d9d640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d8199490 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f46c8005940 tx=0x7f46c80048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:13.429 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f46c803f070 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d819a860 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f46c8003a80 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f46c8035030 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.426+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d819ade0 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.427+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d81027c0 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.428+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f46c8042070 con 0x7f46d8102350 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.429+0000 7f46bffff640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f46c007af30 0x7f46c007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:13.430 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.429+0000 7f46d659c640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f46c007af30 0x7f46c007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.429+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f46c80bea40 con 0x7f46d8102350 2024-09-21T01:28:13.431 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.429+0000 7f46d659c640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f46c007af30 0x7f46c007d440 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0058c0 tx=0x7f46cc00a2d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:13.434 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.433+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f46c808a240 con 0x7f46d8102350 2024-09-21T01:28:13.638 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.637+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f46d80ffeb0 con 0x7f46c007af30 2024-09-21T01:28:13.639 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.638+0000 7f46bffff640 1 -- 172.21.15.45:0/3781076208 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f46d80ffeb0 con 0x7f46c007af30 2024-09-21T01:28:13.639 INFO:teuthology.orchestra.run.smithi045.stdout:Backend: cephadm 2024-09-21T01:28:13.639 INFO:teuthology.orchestra.run.smithi045.stdout:Available: Yes 2024-09-21T01:28:13.639 INFO:teuthology.orchestra.run.smithi045.stdout:Paused: No 2024-09-21T01:28:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f46c007af30 msgr2=0x7f46c007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:13.641 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f46c007af30 0x7f46c007d440 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0058c0 tx=0x7f46cc00a2d0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 msgr2=0x7f46d8199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d8199490 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f46c8005940 tx=0x7f46c80048f0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 shutdown_connections 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f46c007af30 0x7f46c007d440 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46d8103480 0x7f46d8199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 --2- 172.21.15.45:0/3781076208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46d8102350 0x7f46d8199490 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.640+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 >> 172.21.15.45:0/3781076208 conn(0x7f46d80fdce0 msgr2=0x7f46d80ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.641+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 shutdown_connections 2024-09-21T01:28:13.642 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:13.641+0000 7f46dd0f9640 1 -- 172.21.15.45:0/3781076208 wait complete. 2024-09-21T01:28:13.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:13 smithi090 ceph-mon[35142]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:14.124 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:13 smithi045 ceph-mon[28361]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:14.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-21T01:28:14.589 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:14.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:14 smithi090 ceph-mon[35142]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:15.022 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:14 smithi045 ceph-mon[28361]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.047+0000 7f72a42e7640 1 -- 172.21.15.45:0/47174871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 msgr2=0x7f729c106090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.047+0000 7f72a42e7640 1 --2- 172.21.15.45:0/47174871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c106090 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7298009880 tx=0x7f729802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- 172.21.15.45:0/47174871 shutdown_connections 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 --2- 172.21.15.45:0/47174871 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c106620 0x7f729c108b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 --2- 172.21.15.45:0/47174871 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c106090 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- 172.21.15.45:0/47174871 >> 172.21.15.45:0/47174871 conn(0x7f729c0fdcb0 msgr2=0x7f729c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- 172.21.15.45:0/47174871 shutdown_connections 2024-09-21T01:28:15.049 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- 172.21.15.45:0/47174871 wait complete. 2024-09-21T01:28:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 Processor -- start 2024-09-21T01:28:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- start start 2024-09-21T01:28:15.051 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c106620 0x7f729c199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f729c10c250 con 0x7f729c06c2b0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.048+0000 7f72a42e7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f729c10c0d0 con 0x7f729c106620 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c1994f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:48990/0 (socket says 172.21.15.45:48990) 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 -- 172.21.15.45:0/1076623825 learned_addr learned my addr 172.21.15.45:0/1076623825 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a1882640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c106620 0x7f729c199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 -- 172.21.15.45:0/1076623825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c106620 msgr2=0x7f729c199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c106620 0x7f729c199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 -- 172.21.15.45:0/1076623825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f729c19a630 con 0x7f729c06c2b0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a2083640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c1994f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7298000c00 tx=0x7f7298004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f729803f070 con 0x7f729c06c2b0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f729c19a8c0 con 0x7f729c06c2b0 2024-09-21T01:28:15.052 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f7298004440 con 0x7f729c06c2b0 2024-09-21T01:28:15.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.049+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f7298035030 con 0x7f729c06c2b0 2024-09-21T01:28:15.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.050+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f729c19ae40 con 0x7f729c06c2b0 2024-09-21T01:28:15.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.051+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f729c104100 con 0x7f729c06c2b0 2024-09-21T01:28:15.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.051+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f7298042070 con 0x7f729c06c2b0 2024-09-21T01:28:15.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.052+0000 7f72937fe640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f728807af30 0x7f728807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:15.053 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.052+0000 7f72a1882640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f728807af30 0x7f728807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:15.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.052+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f72980beb30 con 0x7f729c06c2b0 2024-09-21T01:28:15.054 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.053+0000 7f72a1882640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f728807af30 0x7f728807d440 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f728c00a810 tx=0x7f728c008040 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:15.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.057+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f729808a290 con 0x7f729c06c2b0 2024-09-21T01:28:15.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.253+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f729c108600 con 0x7f728807af30 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.263+0000 7f72937fe640 1 -- 172.21.15.45:0/1076623825 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3410 (secure 0 0 0) 0x7f729c108600 con 0x7f728807af30 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager.smithi045 smithi045 *:9093,9094 running (4m) 2m ago 5m 18.5M - 0.27.0 11f11916f8cd 9f339d414d51 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-exporter.smithi045 smithi045 *:9926 running (6m) 2m ago 6m 75.1M - 19.3.0-5120-g18432218 ddce307e1f32 cf340a3a58e0 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-exporter.smithi090 smithi090 *:9926 running (5m) 41s ago 5m 72.5M - 19.3.0-5120-g18432218 ddce307e1f32 30eba2728bac 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi045 smithi045 running (6m) 2m ago 6m 6895k - 19.3.0-5120-g18432218 ddce307e1f32 b89b99a257be 2024-09-21T01:28:15.264 INFO:teuthology.orchestra.run.smithi045.stdout:crash.smithi090 smithi090 running (5m) 41s ago 5m 6895k - 19.3.0-5120-g18432218 ddce307e1f32 85980a92715f 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:grafana.smithi045 smithi045 *:3000 running (4m) 2m ago 5m 77.4M - 10.4.8 5aad1d7cf004 71a1ee9c0f6b 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi045.txjudz smithi045 *:9283,8765,8443 running (7m) 2m ago 7m 577M - 19.3.0-5120-g18432218 ddce307e1f32 254eab77649b 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:mgr.smithi090.zsfvyu smithi090 *:8443,9283,8765 running (5m) 41s ago 5m 525M - 19.3.0-5120-g18432218 ddce307e1f32 c97a8777f79d 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi045 smithi045 running (7m) 2m ago 7m 115M 2048M 19.3.0-5120-g18432218 ddce307e1f32 387dd03e82eb 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:mon.smithi090 smithi090 running (4m) 41s ago 4m 102M 2048M 19.3.0-5120-g18432218 ddce307e1f32 a7fe49195229 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi045 smithi045 *:9100 running (6m) 2m ago 6m 9.92M - 1.7.0 72c9c2088986 d2e309b2421a 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter.smithi090 smithi090 *:9100 running (5m) 41s ago 5m 10.0M - 1.7.0 72c9c2088986 c563ad47cb53 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.0 smithi045 running (3m) 2m ago 3m 103M 2961M 19.3.0-5120-g18432218 ddce307e1f32 dbba428d3aba 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.1 smithi045 running (3m) 2m ago 3m 105M 2961M 19.3.0-5120-g18432218 ddce307e1f32 5fa2b9b5a6ce 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.2 smithi045 running (2m) 2m ago 2m 102M 2961M 19.3.0-5120-g18432218 ddce307e1f32 a6a30c7bac12 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.3 smithi045 running (2m) 2m ago 2m 81.4M 2961M 19.3.0-5120-g18432218 ddce307e1f32 42791864b159 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.4 smithi090 running (2m) 41s ago 2m 105M 3728M 19.3.0-5120-g18432218 ddce307e1f32 281d55ea95e9 2024-09-21T01:28:15.265 INFO:teuthology.orchestra.run.smithi045.stdout:osd.5 smithi090 running (97s) 41s ago 97s 103M 3728M 19.3.0-5120-g18432218 ddce307e1f32 658b7398123a 2024-09-21T01:28:15.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.6 smithi090 running (72s) 41s ago 72s 107M 3728M 19.3.0-5120-g18432218 ddce307e1f32 d64824ee31d6 2024-09-21T01:28:15.266 INFO:teuthology.orchestra.run.smithi045.stdout:osd.7 smithi090 running (47s) 41s ago 46s 81.4M 3728M 19.3.0-5120-g18432218 ddce307e1f32 effbd3c3de5d 2024-09-21T01:28:15.266 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus.smithi045 smithi045 *:9095 running (4m) 2m ago 5m 41.9M - 2.51.0 1d3b7f56885b 4867f4d6ecd6 2024-09-21T01:28:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f728807af30 msgr2=0x7f728807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f728807af30 0x7f728807d440 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f728c00a810 tx=0x7f728c008040 comp rx=0 tx=0).stop 2024-09-21T01:28:15.266 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 msgr2=0x7f729c1994f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c1994f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7298000c00 tx=0x7f7298004040 comp rx=0 tx=0).stop 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 shutdown_connections 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f728807af30 0x7f728807d440 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f729c106620 0x7f729c199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 --2- 172.21.15.45:0/1076623825 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f729c06c2b0 0x7f729c1994f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 >> 172.21.15.45:0/1076623825 conn(0x7f729c0fdcb0 msgr2=0x7f729c100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 shutdown_connections 2024-09-21T01:28:15.267 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:15.265+0000 7f72a42e7640 1 -- 172.21.15.45:0/1076623825 wait complete. 2024-09-21T01:28:15.885 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:15 smithi045 ceph-mon[28361]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:15.914 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-21T01:28:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:15 smithi090 ceph-mon[35142]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:16.174 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:16.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 -- 172.21.15.45:0/888663756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 msgr2=0x7f61f4108b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:16.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/888663756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f4108b00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f61e8009880 tx=0x7f61e802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 -- 172.21.15.45:0/888663756 shutdown_connections 2024-09-21T01:28:16.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/888663756 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f4108b00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/888663756 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f406c2b0 0x7f61f4106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.645 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 -- 172.21.15.45:0/888663756 >> 172.21.15.45:0/888663756 conn(0x7f61f40fdcb0 msgr2=0x7f61f4100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.624+0000 7f61fa8f3640 1 -- 172.21.15.45:0/888663756 shutdown_connections 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.625+0000 7f61fa8f3640 1 -- 172.21.15.45:0/888663756 wait complete. 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.625+0000 7f61fa8f3640 1 Processor -- start 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.625+0000 7f61fa8f3640 1 -- start start 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61fa8f3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f406c2b0 0x7f61f419b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61fa8f3640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61fa8f3640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f61f410c250 con 0x7f61f4106620 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61fa8f3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f61f410c0d0 con 0x7f61f406c2b0 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f419bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46960/0 (socket says 172.21.15.45:46960) 2024-09-21T01:28:16.646 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 -- 172.21.15.45:0/2403839876 learned_addr learned my addr 172.21.15.45:0/2403839876 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f3fff640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f406c2b0 0x7f61f419b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 -- 172.21.15.45:0/2403839876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f406c2b0 msgr2=0x7f61f419b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f406c2b0 0x7f61f419b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 -- 172.21.15.45:0/2403839876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f419c860 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f37fe640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f419bd90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f61e8000c00 tx=0x7f61e8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.626+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f61e803f070 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.627+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61f419caf0 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.627+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f61e8004440 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.627+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f61e8035030 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.627+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61f419d070 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.628+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61f4104100 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.629+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f61e8042070 con 0x7f61f4106620 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.629+0000 7f61f17fa640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f61dc07af30 0x7f61dc07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:16.647 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.629+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7f61e80bea90 con 0x7f61f4106620 2024-09-21T01:28:16.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.629+0000 7f61f3fff640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f61dc07af30 0x7f61dc07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:16.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.630+0000 7f61f3fff640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f61dc07af30 0x7f61dc07d440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61e400a810 tx=0x7f61e4008040 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:16.648 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.634+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f61e808a290 con 0x7f61f4106620 2024-09-21T01:28:16.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.831+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f61f4102f40 con 0x7f61dc07af30 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.836+0000 7f61f17fa640 1 -- 172.21.15.45:0/2403839876 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+1290 (secure 0 0 0) 0x7f61f4102f40 con 0x7f61dc07af30 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-exporter ?:9926 2/2 2m ago 6m * 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:crash 2/2 2m ago 6m * 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:mgr 2/2 2m ago 6m count:2 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:mon 2/2 2m ago 6m smithi045:172.21.15.45=smithi045;smithi090:172.21.15.90=smithi090;count:2 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-09-21T01:28:16.837 INFO:teuthology.orchestra.run.smithi045.stdout:osd 8 2m ago - 2024-09-21T01:28:16.838 INFO:teuthology.orchestra.run.smithi045.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f61dc07af30 msgr2=0x7f61dc07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f61dc07af30 0x7f61dc07d440 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f61e400a810 tx=0x7f61e4008040 comp rx=0 tx=0).stop 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 msgr2=0x7f61f419bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f419bd90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f61e8000c00 tx=0x7f61e8004040 comp rx=0 tx=0).stop 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 shutdown_connections 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f61dc07af30 0x7f61dc07d440 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f61f4106620 0x7f61f419bd90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 --2- 172.21.15.45:0/2403839876 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f61f406c2b0 0x7f61f419b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 >> 172.21.15.45:0/2403839876 conn(0x7f61f40fdcb0 msgr2=0x7f61f4102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 shutdown_connections 2024-09-21T01:28:16.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:16.838+0000 7f61fa8f3640 1 -- 172.21.15.45:0/2403839876 wait complete. 2024-09-21T01:28:16.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:16 smithi045 ceph-mon[28361]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:16.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:16 smithi090 ceph-mon[35142]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:17.497 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-21T01:28:17.764 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:17 smithi090 ceph-mon[35142]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:18.002 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:17 smithi045 ceph-mon[28361]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 -- 172.21.15.45:0/100925240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 msgr2=0x7fe5801027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 --2- 172.21.15.45:0/100925240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe5801027b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009880 tx=0x7fe57c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 -- 172.21.15.45:0/100925240 shutdown_connections 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 --2- 172.21.15.45:0/100925240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe580103470 0x7fe580103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 --2- 172.21.15.45:0/100925240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe5801027b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 -- 172.21.15.45:0/100925240 >> 172.21.15.45:0/100925240 conn(0x7fe5800fdcd0 msgr2=0x7fe580100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 -- 172.21.15.45:0/100925240 shutdown_connections 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.249+0000 7fe587eb1640 1 -- 172.21.15.45:0/100925240 wait complete. 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe587eb1640 1 Processor -- start 2024-09-21T01:28:18.251 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe587eb1640 1 -- start start 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe587eb1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe580199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe587eb1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe580103470 0x7fe580199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe587eb1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe58010b0e0 con 0x7fe580102340 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe587eb1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe58010af60 con 0x7fe580103470 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe585c4d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe580199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe585c4d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe580199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46984/0 (socket says 172.21.15.45:46984) 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe585c4d640 1 -- 172.21.15.45:0/1476101151 learned_addr learned my addr 172.21.15.45:0/1476101151 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.250+0000 7fe58544c640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe580103470 0x7fe580199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe585c4d640 1 -- 172.21.15.45:0/1476101151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe580103470 msgr2=0x7fe580199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe585c4d640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe580103470 0x7fe580199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe585c4d640 1 -- 172.21.15.45:0/1476101151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58019a550 con 0x7fe580102340 2024-09-21T01:28:18.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe585c4d640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe580199410 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe57c000c00 tx=0x7fe57c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe57c03f070 con 0x7fe580102340 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe57c004440 con 0x7fe580102340 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe57c035030 con 0x7fe580102340 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58019a7e0 con 0x7fe580102340 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.251+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58019ad60 con 0x7fe580102340 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.252+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5801027b0 con 0x7fe580102340 2024-09-21T01:28:18.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.253+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fe57c042070 con 0x7fe580102340 2024-09-21T01:28:18.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.254+0000 7fe576ffd640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe56c07af80 0x7fe56c07d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:18.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.254+0000 7fe58544c640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe56c07af80 0x7fe56c07d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:18.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.254+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fe57c0bed40 con 0x7fe580102340 2024-09-21T01:28:18.257 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.254+0000 7fe58544c640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe56c07af80 0x7fe56c07d490 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe57000abe0 tx=0x7fe57000a2d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:18.259 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.258+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fe57c08a660 con 0x7fe580102340 2024-09-21T01:28:18.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.450+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fe5800ffe10 con 0x7fe56c07af80 2024-09-21T01:28:18.453 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.452+0000 7fe576ffd640 1 -- 172.21.15.45:0/1476101151 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+145 (secure 0 0 0) 0x7fe5800ffe10 con 0x7fe56c07af80 2024-09-21T01:28:18.453 INFO:teuthology.orchestra.run.smithi045.stdout:HOST ADDR LABELS STATUS 2024-09-21T01:28:18.453 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 172.21.15.45 2024-09-21T01:28:18.453 INFO:teuthology.orchestra.run.smithi045.stdout:smithi090 172.21.15.90 2024-09-21T01:28:18.453 INFO:teuthology.orchestra.run.smithi045.stdout:2 hosts in cluster 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe56c07af80 msgr2=0x7fe56c07d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe56c07af80 0x7fe56c07d490 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe57000abe0 tx=0x7fe57000a2d0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 msgr2=0x7fe580199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe580199410 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe57c000c00 tx=0x7fe57c004040 comp rx=0 tx=0).stop 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 shutdown_connections 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe56c07af80 0x7fe56c07d490 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe580103470 0x7fe580199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 --2- 172.21.15.45:0/1476101151 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe580102340 0x7fe580199410 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 >> 172.21.15.45:0/1476101151 conn(0x7fe5800fdcd0 msgr2=0x7fe5800ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:18.456 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 shutdown_connections 2024-09-21T01:28:18.457 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:18.454+0000 7fe587eb1640 1 -- 172.21.15.45:0/1476101151 wait complete. 2024-09-21T01:28:19.060 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:18 smithi045 ceph-mon[28361]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:19.092 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-21T01:28:19.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:18 smithi090 ceph-mon[35142]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:19.354 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:19.786 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 -- 172.21.15.45:0/2835555389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798103470 msgr2=0x7ff798103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 --2- 172.21.15.45:0/2835555389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798103470 0x7ff798103880 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009880 tx=0x7ff78c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 -- 172.21.15.45:0/2835555389 shutdown_connections 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 --2- 172.21.15.45:0/2835555389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798103470 0x7ff798103880 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 --2- 172.21.15.45:0/2835555389 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff798102340 0x7ff7981027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 -- 172.21.15.45:0/2835555389 >> 172.21.15.45:0/2835555389 conn(0x7ff7980fdcb0 msgr2=0x7ff798100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 -- 172.21.15.45:0/2835555389 shutdown_connections 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.783+0000 7ff79ea84640 1 -- 172.21.15.45:0/2835555389 wait complete. 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79ea84640 1 Processor -- start 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79ea84640 1 -- start start 2024-09-21T01:28:19.787 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79ea84640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 0x7ff798199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79ea84640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff798103470 0x7ff798199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79ea84640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff79810b0f0 con 0x7ff798102340 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79ea84640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff79810af70 con 0x7ff798103470 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79c820640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 0x7ff798199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79c820640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 0x7ff798199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46990/0 (socket says 172.21.15.45:46990) 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79c820640 1 -- 172.21.15.45:0/3760575784 learned_addr learned my addr 172.21.15.45:0/3760575784 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff797fff640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff798103470 0x7ff798199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79c820640 1 -- 172.21.15.45:0/3760575784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff798103470 msgr2=0x7ff798199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79c820640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff798103470 0x7ff798199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.784+0000 7ff79c820640 1 -- 172.21.15.45:0/3760575784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff79819a5a0 con 0x7ff798102340 2024-09-21T01:28:19.788 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.785+0000 7ff79c820640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 0x7ff798199460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c8a0 tx=0x7ff78800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.785+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff7880079b0 con 0x7ff798102340 2024-09-21T01:28:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.785+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff78800f450 con 0x7ff798102340 2024-09-21T01:28:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.785+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff79819a890 con 0x7ff798102340 2024-09-21T01:28:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.785+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff788018630 con 0x7ff798102340 2024-09-21T01:28:19.789 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.785+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff79819ae70 con 0x7ff798102340 2024-09-21T01:28:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.786+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff79810b840 con 0x7ff798102340 2024-09-21T01:28:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.787+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7ff788007b50 con 0x7ff798102340 2024-09-21T01:28:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.787+0000 7ff795ffb640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff78007af30 0x7ff78007d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.787+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7ff78809e030 con 0x7ff798102340 2024-09-21T01:28:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.787+0000 7ff797fff640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff78007af30 0x7ff78007d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:19.790 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.788+0000 7ff797fff640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff78007af30 0x7ff78007d440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff78c004d10 tx=0x7ff78c005d50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:19.793 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.792+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff788014030 con 0x7ff798102340 2024-09-21T01:28:19.986 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.985+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff79806bef0 con 0x7ff78007af30 2024-09-21T01:28:19.988 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.987+0000 7ff795ffb640 1 -- 172.21.15.45:0/3760575784 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+399 (secure 0 0 0) 0x7ff79806bef0 con 0x7ff78007af30 2024-09-21T01:28:19.988 INFO:teuthology.orchestra.run.smithi045.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-21T01:28:19.988 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75120098480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-09-21T01:28:19.988 INFO:teuthology.orchestra.run.smithi045.stdout:smithi090 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UK400BGN 372G No 43s ago Has a FileSystem, LVM detected 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff78007af30 msgr2=0x7ff78007d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff78007af30 0x7ff78007d440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff78c004d10 tx=0x7ff78c005d50 comp rx=0 tx=0).stop 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 msgr2=0x7ff798199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 0x7ff798199460 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff78800c8a0 tx=0x7ff78800cd60 comp rx=0 tx=0).stop 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 shutdown_connections 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff78007af30 0x7ff78007d440 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.990 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff798103470 0x7ff798199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 --2- 172.21.15.45:0/3760575784 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff798102340 0x7ff798199460 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 >> 172.21.15.45:0/3760575784 conn(0x7ff7980fdcb0 msgr2=0x7ff79806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 shutdown_connections 2024-09-21T01:28:19.991 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:19.989+0000 7ff79ea84640 1 -- 172.21.15.45:0/3760575784 wait complete. 2024-09-21T01:28:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:19 smithi045 ceph-mon[28361]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:19 smithi045 ceph-mon[28361]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:19 smithi090 ceph-mon[35142]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:19 smithi090 ceph-mon[35142]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:20.553 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T01:28:20.562 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi045.front.sepia.ceph.com 2024-09-21T01:28:20.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph fs volume create foo' 2024-09-21T01:28:20.820 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.262+0000 7fdbe0b63640 1 -- 172.21.15.45:0/4055233463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 msgr2=0x7fdbdc103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:21.269 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.262+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/4055233463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc103880 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4009880 tx=0x7fdbc402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.263+0000 7fdbe0b63640 1 -- 172.21.15.45:0/4055233463 shutdown_connections 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.263+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/4055233463 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc103880 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.263+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/4055233463 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbdc102340 0x7fdbdc1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.263+0000 7fdbe0b63640 1 -- 172.21.15.45:0/4055233463 >> 172.21.15.45:0/4055233463 conn(0x7fdbdc0fdcb0 msgr2=0x7fdbdc100160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.263+0000 7fdbe0b63640 1 -- 172.21.15.45:0/4055233463 shutdown_connections 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.263+0000 7fdbe0b63640 1 -- 172.21.15.45:0/4055233463 wait complete. 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbe0b63640 1 Processor -- start 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbe0b63640 1 -- start start 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbe0b63640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbdc102340 0x7fdbdc1993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbe0b63640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbe0b63640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc10c1e0 con 0x7fdbdc103470 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbe0b63640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdbdc10c060 con 0x7fdbdc102340 2024-09-21T01:28:21.270 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbd9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbd9d9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47000/0 (socket says 172.21.15.45:47000) 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbd9d9b640 1 -- 172.21.15.45:0/2756197629 learned_addr learned my addr 172.21.15.45:0/2756197629 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.264+0000 7fdbda59c640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbdc102340 0x7fdbdc1993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd9d9b640 1 -- 172.21.15.45:0/2756197629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbdc102340 msgr2=0x7fdbdc1993e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd9d9b640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbdc102340 0x7fdbdc1993e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd9d9b640 1 -- 172.21.15.45:0/2756197629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbdc19a520 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd9d9b640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc199a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4000c00 tx=0x7fdbc4004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdbc403f070 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdbc4004440 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdbc4035030 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbdc19a7b0 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.265+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbdc19ad30 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.266+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbdc1027b0 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.267+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fdbc4042070 con 0x7fdbdc103470 2024-09-21T01:28:21.271 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.268+0000 7fdbd37fe640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdbc807af80 0x7fdbc807d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:21.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.268+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5502+0+0 (secure 0 0 0) 0x7fdbc40bea90 con 0x7fdbdc103470 2024-09-21T01:28:21.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.268+0000 7fdbda59c640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdbc807af80 0x7fdbc807d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:21.272 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.268+0000 7fdbda59c640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdbc807af80 0x7fdbc807d490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0058c0 tx=0x7fdbcc00a4c0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:21.273 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.272+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdbc408a270 con 0x7fdbdc103470 2024-09-21T01:28:21.485 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:21.484+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}) -- 0x7fdbdc107f20 con 0x7fdbc807af80 2024-09-21T01:28:22.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:21 smithi045 ceph-mon[28361]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:22.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:21 smithi045 ceph-mon[28361]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:22.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:21 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool create", "pool": "cephfs.foo.meta"} : dispatch 2024-09-21T01:28:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:21 smithi090 ceph-mon[35142]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:21 smithi090 ceph-mon[35142]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:22.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:21 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool create", "pool": "cephfs.foo.meta"} : dispatch 2024-09-21T01:28:22.796 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.796+0000 7fdbd37fe640 1 -- 172.21.15.45:0/2756197629 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fdbdc107f20 con 0x7fdbc807af80 2024-09-21T01:28:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdbc807af80 msgr2=0x7fdbc807d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdbc807af80 0x7fdbc807d490 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0058c0 tx=0x7fdbcc00a4c0 comp rx=0 tx=0).stop 2024-09-21T01:28:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 msgr2=0x7fdbdc199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc199a50 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4000c00 tx=0x7fdbc4004040 comp rx=0 tx=0).stop 2024-09-21T01:28:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 shutdown_connections 2024-09-21T01:28:22.800 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdbc807af80 0x7fdbc807d490 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc0058c0 tx=0x7fdbcc00a4c0 comp rx=0 tx=0).stop 2024-09-21T01:28:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdbdc103470 0x7fdbdc199a50 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 --2- 172.21.15.45:0/2756197629 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdbdc102340 0x7fdbdc1993e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 >> 172.21.15.45:0/2756197629 conn(0x7fdbdc0fdcb0 msgr2=0x7fdbdc106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 shutdown_connections 2024-09-21T01:28:22.801 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:22.799+0000 7fdbe0b63640 1 -- 172.21.15.45:0/2756197629 wait complete. 2024-09-21T01:28:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:22 smithi045 ceph-mon[28361]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T01:28:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:22 smithi045 ceph-mon[28361]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T01:28:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"} : dispatch 2024-09-21T01:28:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:22.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:22 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045[28357]: 2024-09-21T01:28:22.773+0000 7fa5ec64c640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:28:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:22 smithi090 ceph-mon[35142]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T01:28:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:22 smithi090 ceph-mon[35142]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T01:28:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"} : dispatch 2024-09-21T01:28:23.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:23.542 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- bash -c 'ceph fs volume create bar' 2024-09-21T01:28:24.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: pgmap v206: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:24.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T01:28:24.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T01:28:24.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"} : dispatch 2024-09-21T01:28:24.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:28:24.027 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: fsmap foo:0 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: Saving service mds.foo spec with placement count:2 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi045.qpbfpk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi045.qpbfpk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:24.028 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: pgmap v206: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"} : dispatch 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: fsmap foo:0 2024-09-21T01:28:24.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: Saving service mds.foo spec with placement count:2 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi045.qpbfpk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi045.qpbfpk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:24.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:24.935 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:25.061 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:24 smithi045 ceph-mon[28361]: Deploying daemon mds.foo.smithi045.qpbfpk on smithi045 2024-09-21T01:28:25.061 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:24 smithi045 ceph-mon[28361]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T01:28:25.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:24 smithi090 ceph-mon[35142]: Deploying daemon mds.foo.smithi045.qpbfpk on smithi045 2024-09-21T01:28:25.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:24 smithi090 ceph-mon[35142]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T01:28:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 -- 172.21.15.45:0/3326422357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090102360 msgr2=0x7fa0901027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 --2- 172.21.15.45:0/3326422357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090102360 0x7fa0901027d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa08c009880 tx=0x7fa08c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 -- 172.21.15.45:0/3326422357 shutdown_connections 2024-09-21T01:28:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 --2- 172.21.15.45:0/3326422357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa090103490 0x7fa0901038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 --2- 172.21.15.45:0/3326422357 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090102360 0x7fa0901027d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:25.585 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 -- 172.21.15.45:0/3326422357 >> 172.21.15.45:0/3326422357 conn(0x7fa0900fdcd0 msgr2=0x7fa090100180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 -- 172.21.15.45:0/3326422357 shutdown_connections 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.582+0000 7fa098256640 1 -- 172.21.15.45:0/3326422357 wait complete. 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa098256640 1 Processor -- start 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa098256640 1 -- start start 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa098256640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa090102360 0x7fa090199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa098256640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 0x7fa090199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa098256640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa09010b180 con 0x7fa090103490 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa098256640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa09010b000 con 0x7fa090102360 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.583+0000 7fa0957f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 0x7fa090199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa0957f1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 0x7fa090199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:47018/0 (socket says 172.21.15.45:47018) 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa0957f1640 1 -- 172.21.15.45:0/3330598705 learned_addr learned my addr 172.21.15.45:0/3330598705 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa095ff2640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa090102360 0x7fa090199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:25.586 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa0957f1640 1 -- 172.21.15.45:0/3330598705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa090102360 msgr2=0x7fa090199570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa0957f1640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa090102360 0x7fa090199570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa0957f1640 1 -- 172.21.15.45:0/3330598705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09019a6b0 con 0x7fa090103490 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa0957f1640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 0x7fa090199be0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa08000b420 tx=0x7fa08000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa080011630 con 0x7fa090103490 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa080016070 con 0x7fa090103490 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa09019a9a0 con 0x7fa090103490 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa080010b80 con 0x7fa090103490 2024-09-21T01:28:25.587 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.584+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa09019af80 con 0x7fa090103490 2024-09-21T01:28:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.586+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa09010b8d0 con 0x7fa090103490 2024-09-21T01:28:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.587+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fa0800117d0 con 0x7fa090103490 2024-09-21T01:28:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.587+0000 7fa086ffd640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa07c07af30 0x7fa07c07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.587+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 6320+0+0 (secure 0 0 0) 0x7fa08009d9a0 con 0x7fa090103490 2024-09-21T01:28:25.588 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.587+0000 7fa095ff2640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa07c07af30 0x7fa07c07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:25.589 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.588+0000 7fa095ff2640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa07c07af30 0x7fa07c07d440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa08c004d10 tx=0x7fa08c004d70 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:25.592 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.592+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa080068e70 con 0x7fa090103490 2024-09-21T01:28:25.823 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:25.823+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 --> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}) -- 0x7fa0900ffec0 con 0x7fa07c07af30 2024-09-21T01:28:26.063 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:25 smithi045 ceph-mon[28361]: pgmap v210: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:26.063 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:25 smithi045 ceph-mon[28361]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T01:28:26.186 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:25 smithi090 ceph-mon[35142]: pgmap v210: 65 pgs: 9 creating+peering, 55 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:26.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:25 smithi090 ceph-mon[35142]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T01:28:26.917 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool create", "pool": "cephfs.bar.meta"} : dispatch 2024-09-21T01:28:26.917 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:26.917 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:26.917 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:26.918 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.savegz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:26.918 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.savegz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:26.918 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:26 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pool create", "pool": "cephfs.bar.meta"} : dispatch 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.savegz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.savegz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:27.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:26 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:27.893 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:27.893 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: pgmap v212: 65 pgs: 9 creating+peering, 21 unknown, 35 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:27.893 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: Deploying daemon mds.foo.smithi090.savegz on smithi090 2024-09-21T01:28:27.893 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]': finished 2024-09-21T01:28:27.893 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] up:boot 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: daemon mds.foo.smithi045.qpbfpk assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: Cluster is now healthy 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: fsmap foo:0 1 up:standby 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "foo.smithi045.qpbfpk"} : dispatch 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: fsmap foo:1 {0=foo.smithi045.qpbfpk=up:creating} 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"} : dispatch 2024-09-21T01:28:27.894 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:27 smithi090 ceph-mon[35142]: daemon mds.foo.smithi045.qpbfpk is now active in filesystem foo as rank 0 2024-09-21T01:28:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: from='client.14616 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T01:28:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: pgmap v212: 65 pgs: 9 creating+peering, 21 unknown, 35 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:28:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: Deploying daemon mds.foo.smithi090.savegz on smithi090 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]': finished 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] up:boot 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: daemon mds.foo.smithi045.qpbfpk assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: Cluster is now healthy 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: fsmap foo:0 1 up:standby 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "foo.smithi045.qpbfpk"} : dispatch 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: fsmap foo:1 {0=foo.smithi045.qpbfpk=up:creating} 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"} : dispatch 2024-09-21T01:28:28.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:27 smithi045 ceph-mon[28361]: daemon mds.foo.smithi045.qpbfpk is now active in filesystem foo as rank 0 2024-09-21T01:28:28.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.833+0000 7fa086ffd640 1 -- 172.21.15.45:0/3330598705 <== mgr.14227 v2:172.21.15.45:6800/3847851980 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa0900ffec0 con 0x7fa07c07af30 2024-09-21T01:28:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.837+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa07c07af30 msgr2=0x7fa07c07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.837+0000 7fa098256640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa07c07af30 0x7fa07c07d440 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa08c004d10 tx=0x7fa08c004d70 comp rx=0 tx=0).stop 2024-09-21T01:28:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.837+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 msgr2=0x7fa090199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.837+0000 7fa098256640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 0x7fa090199be0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa08000b420 tx=0x7fa08000b8e0 comp rx=0 tx=0).stop 2024-09-21T01:28:28.838 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 shutdown_connections 2024-09-21T01:28:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa07c07af30 0x7fa07c07d440 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa090103490 0x7fa090199be0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 --2- 172.21.15.45:0/3330598705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa090102360 0x7fa090199570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 >> 172.21.15.45:0/3330598705 conn(0x7fa0900fdcd0 msgr2=0x7fa0900ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 shutdown_connections 2024-09-21T01:28:28.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:28.838+0000 7fa098256640 1 -- 172.21.15.45:0/3330598705 wait complete. 2024-09-21T01:28:28.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]': finished 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"} : dispatch 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] up:active 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: fsmap foo:1 {0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:28.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]': finished 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"} : dispatch 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] up:active 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: fsmap foo:1 {0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:29.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:29.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:29.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:28 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045[28357]: 2024-09-21T01:28:28.814+0000 7fa5eee51640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:28:29.569 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-21T01:28:29.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: pgmap v216: 129 pgs: 12 creating+peering, 52 unknown, 65 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 1 op/s 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]': finished 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.90:6832/3876994677,v1:172.21.15.90:6833/3876994677] up:boot 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: daemon mds.foo.smithi090.savegz assigned to filesystem bar as rank 0 (now has 1 ranks) 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:28:29.832 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: fsmap foo:1 bar:0 {foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:29.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "foo.smithi090.savegz"} : dispatch 2024-09-21T01:28:29.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:creating,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:29.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:29.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T01:28:29.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:29 smithi090 ceph-mon[35142]: daemon mds.foo.smithi090.savegz is now active in filesystem bar as rank 0 2024-09-21T01:28:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: pgmap v216: 129 pgs: 12 creating+peering, 52 unknown, 65 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 1 op/s 2024-09-21T01:28:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T01:28:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T01:28:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]': finished 2024-09-21T01:28:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.90:6832/3876994677,v1:172.21.15.90:6833/3876994677] up:boot 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: daemon mds.foo.smithi090.savegz assigned to filesystem bar as rank 0 (now has 1 ranks) 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: fsmap foo:1 bar:0 {foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "foo.smithi090.savegz"} : dispatch 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:creating,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T01:28:30.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:29 smithi045 ceph-mon[28361]: daemon mds.foo.smithi090.savegz is now active in filesystem bar as rank 0 2024-09-21T01:28:30.436 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:30.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:30 smithi045 ceph-mon[28361]: Saving service mds.bar spec with placement count:2 2024-09-21T01:28:30.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:30 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.90:6832/3876994677,v1:172.21.15.90:6833/3876994677] up:active 2024-09-21T01:28:30.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:30 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:30.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:30 smithi045 ceph-mon[28361]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T01:28:30.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:30.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:31.083 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:30 smithi090 ceph-mon[35142]: Saving service mds.bar spec with placement count:2 2024-09-21T01:28:31.083 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:30 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.90:6832/3876994677,v1:172.21.15.90:6833/3876994677] up:active 2024-09-21T01:28:31.083 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:30 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:31.083 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:30 smithi090 ceph-mon[35142]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T01:28:31.083 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:31.083 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 -- 172.21.15.45:0/3660756675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 msgr2=0x7faac01095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 --2- 172.21.15.45:0/3660756675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac01095a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009880 tx=0x7faaa802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 -- 172.21.15.45:0/3660756675 shutdown_connections 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 --2- 172.21.15.45:0/3660756675 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac01095a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 --2- 172.21.15.45:0/3660756675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 0x7faac0103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 -- 172.21.15.45:0/3660756675 >> 172.21.15.45:0/3660756675 conn(0x7faac00ff2a0 msgr2=0x7faac01016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 -- 172.21.15.45:0/3660756675 shutdown_connections 2024-09-21T01:28:31.276 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.209+0000 7faac5b6e640 1 -- 172.21.15.45:0/3660756675 wait complete. 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.210+0000 7faac5b6e640 1 Processor -- start 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.210+0000 7faac5b6e640 1 -- start start 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.210+0000 7faac5b6e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 0x7faac019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.210+0000 7faac5b6e640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac019c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.210+0000 7faac5b6e640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7faac01108e0 con 0x7faac0109190 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.210+0000 7faac5b6e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7faac0110760 con 0x7faac0103850 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac019c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac019c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52866/0 (socket says 172.21.15.45:52866) 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 -- 172.21.15.45:0/4012728353 learned_addr learned my addr 172.21.15.45:0/4012728353 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faabf7fe640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 0x7faac019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 -- 172.21.15.45:0/4012728353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 msgr2=0x7faac019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 0x7faac019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 -- 172.21.15.45:0/4012728353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac01a12f0 con 0x7faac0109190 2024-09-21T01:28:31.277 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faabf7fe640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 0x7faac019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faab6ffd640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac019c050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7faaa802a9c0 tx=0x7faaa80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7faaa803f070 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7faaa80048d0 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faac01a1580 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7faaa8035030 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.211+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faac01a1b30 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.212+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac0104ad0 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.213+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7faaa8042070 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.214+0000 7faabd7fa640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7faaac07af80 0x7faaac07d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.214+0000 7faabf7fe640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7faaac07af80 0x7faaac07d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.214+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7138+0+0 (secure 0 0 0) 0x7faaa80bedb0 con 0x7faac0109190 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.214+0000 7faabf7fe640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7faaac07af80 0x7faaac07d490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faab0009c20 tx=0x7faab0005f90 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:31.278 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.219+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7faaa8089fc0 con 0x7faac0109190 2024-09-21T01:28:31.447 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.447+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faac01024e0 con 0x7faac0109190 2024-09-21T01:28:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.448+0000 7faabd7fa640 1 -- 172.21.15.45:0/4012728353 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+4924 (secure 0 0 0) 0x7faaa808ee70 con 0x7faac0109190 2024-09-21T01:28:31.448 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:31.448 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":8,"btime":"2024-09-21T01:28:29:837028+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":5,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:22.773074+0000","modified":"2024-09-21T01:28:27.802694+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14620},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14620":{"gid":14620,"name":"foo.smithi045.qpbfpk","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.45:6835/4020685920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4020685920},{"type":"v1","addr":"172.21.15.45:6835","nonce":4020685920}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14620,"qdb_cluster":[14620]},"id":1},{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:27.820077+0000","modified":"2024-09-21T01:28:29.837023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24367},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24367":{"gid":24367,"name":"foo.smithi090.savegz","rank":0,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.90:6833/3876994677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3876994677},{"type":"v1","addr":"172.21.15.90:6833","nonce":3876994677}]},"join_fscid":1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24367,"qdb_cluster":[24367]},"id":2}]} 2024-09-21T01:28:31.448 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 8 2024-09-21T01:28:31.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7faaac07af80 msgr2=0x7faaac07d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:31.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7faaac07af80 0x7faaac07d490 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faab0009c20 tx=0x7faab0005f90 comp rx=0 tx=0).stop 2024-09-21T01:28:31.450 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 msgr2=0x7faac019c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac019c050 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7faaa802a9c0 tx=0x7faaa80045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 shutdown_connections 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7faaac07af80 0x7faaac07d490 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faac0109190 0x7faac019c050 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 --2- 172.21.15.45:0/4012728353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faac0103850 0x7faac019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 >> 172.21.15.45:0/4012728353 conn(0x7faac00ff2a0 msgr2=0x7faac00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 shutdown_connections 2024-09-21T01:28:31.451 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:31.450+0000 7faac5b6e640 1 -- 172.21.15.45:0/4012728353 wait complete. 2024-09-21T01:28:32.148 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:31 smithi045 ceph-mon[28361]: pgmap v219: 129 pgs: 12 creating+peering, 39 unknown, 78 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s wr, 4 op/s 2024-09-21T01:28:32.148 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:31 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4012728353' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:32.178 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:31 smithi090 ceph-mon[35142]: pgmap v219: 129 pgs: 12 creating+peering, 39 unknown, 78 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s wr, 4 op/s 2024-09-21T01:28:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:31 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4012728353' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:33.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:32 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] up:active 2024-09-21T01:28:33.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:32 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:33.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:32 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:33.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:32 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] up:active 2024-09-21T01:28:33.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:32 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:33.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:32 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:33.639 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:34.025 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:33 smithi045 ceph-mon[28361]: pgmap v220: 129 pgs: 12 creating+peering, 8 unknown, 109 active+clean; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s wr, 13 op/s 2024-09-21T01:28:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:33 smithi090 ceph-mon[35142]: pgmap v220: 129 pgs: 12 creating+peering, 8 unknown, 109 active+clean; 580 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s wr, 13 op/s 2024-09-21T01:28:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 -- 172.21.15.45:0/954457451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc701046a0 msgr2=0x7fdc70104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 --2- 172.21.15.45:0/954457451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc701046a0 0x7fdc70104ab0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c009880 tx=0x7fdc6c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 -- 172.21.15.45:0/954457451 shutdown_connections 2024-09-21T01:28:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 --2- 172.21.15.45:0/954457451 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc70104ff0 0x7fdc7010d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 --2- 172.21.15.45:0/954457451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc701046a0 0x7fdc70104ab0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 -- 172.21.15.45:0/954457451 >> 172.21.15.45:0/954457451 conn(0x7fdc700fe1a0 msgr2=0x7fdc701005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.513+0000 7fdc77878640 1 -- 172.21.15.45:0/954457451 shutdown_connections 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 -- 172.21.15.45:0/954457451 wait complete. 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 Processor -- start 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 -- start start 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc701046a0 0x7fdc70199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 0x7fdc70199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fdc70110910 con 0x7fdc70104ff0 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.514+0000 7fdc77878640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fdc70110790 con 0x7fdc701046a0 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 0x7fdc70199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc75614640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc701046a0 0x7fdc70199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 0x7fdc70199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52882/0 (socket says 172.21.15.45:52882) 2024-09-21T01:28:34.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 -- 172.21.15.45:0/442017909 learned_addr learned my addr 172.21.15.45:0/442017909 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 -- 172.21.15.45:0/442017909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc701046a0 msgr2=0x7fdc70199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc701046a0 0x7fdc70199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 -- 172.21.15.45:0/442017909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7019f150 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc74e13640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 0x7fdc70199eb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000d6d0 tx=0x7fdc6000db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdc60017070 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fdc600054b0 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc7019f440 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fdc600049d0 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.515+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc7019f9d0 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.516+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7006b290 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.517+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fdc60009940 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.518+0000 7fdc667fc640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc5c07af60 0x7fdc5c07d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.518+0000 7fdc75614640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc5c07af60 0x7fdc5c07d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.518+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7138+0+0 (secure 0 0 0) 0x7fdc6009e160 con 0x7fdc70104ff0 2024-09-21T01:28:34.523 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.518+0000 7fdc75614640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc5c07af60 0x7fdc5c07d470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c000c00 tx=0x7fdc6c0058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:34.524 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.523+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fdc600692f0 con 0x7fdc70104ff0 2024-09-21T01:28:34.749 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.749+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc7019fc80 con 0x7fdc70104ff0 2024-09-21T01:28:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.750+0000 7fdc667fc640 1 -- 172.21.15.45:0/442017909 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+4923 (secure 0 0 0) 0x7fdc6006e1a0 con 0x7fdc70104ff0 2024-09-21T01:28:34.750 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:34.750 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":9,"btime":"2024-09-21T01:28:31:807791+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:22.773074+0000","modified":"2024-09-21T01:28:30.825886+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14620},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14620":{"gid":14620,"name":"foo.smithi045.qpbfpk","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/4020685920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4020685920},{"type":"v1","addr":"172.21.15.45:6835","nonce":4020685920}]},"join_fscid":1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14620,"qdb_cluster":[14620]},"id":1},{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:27.820077+0000","modified":"2024-09-21T01:28:29.837023+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24367},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24367":{"gid":24367,"name":"foo.smithi090.savegz","rank":0,"incarnation":7,"state":"up:active","state_seq":2,"addr":"172.21.15.90:6833/3876994677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":3876994677},{"type":"v1","addr":"172.21.15.90:6833","nonce":3876994677}]},"join_fscid":1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24367,"qdb_cluster":[24367]},"id":2}]} 2024-09-21T01:28:34.750 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 9 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc5c07af60 msgr2=0x7fdc5c07d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc5c07af60 0x7fdc5c07d470 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c000c00 tx=0x7fdc6c0058d0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 msgr2=0x7fdc70199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 0x7fdc70199eb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000d6d0 tx=0x7fdc6000db90 comp rx=0 tx=0).stop 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 shutdown_connections 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fdc5c07af60 0x7fdc5c07d470 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.752 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc70104ff0 0x7fdc70199eb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 --2- 172.21.15.45:0/442017909 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdc701046a0 0x7fdc70199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:34.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 >> 172.21.15.45:0/442017909 conn(0x7fdc700fe1a0 msgr2=0x7fdc7010ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:34.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 shutdown_connections 2024-09-21T01:28:34.753 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:34.752+0000 7fdc77878640 1 -- 172.21.15.45:0/442017909 wait complete. 2024-09-21T01:28:34.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:34 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/442017909' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:35.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:34 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/442017909' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:35.488 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:28:35.766 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: pgmap v221: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s wr, 13 op/s 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.bar.smithi045.tspicn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi045.tspicn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:36.146 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:36.147 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:35 smithi045 ceph-mon[28361]: Deploying daemon mds.bar.smithi045.tspicn on smithi045 2024-09-21T01:28:36.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: pgmap v221: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s wr, 13 op/s 2024-09-21T01:28:36.187 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.bar.smithi045.tspicn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi045.tspicn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:36.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:35 smithi090 ceph-mon[35142]: Deploying daemon mds.bar.smithi045.tspicn on smithi045 2024-09-21T01:28:36.253 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 -- 172.21.15.45:0/2444494165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 msgr2=0x7f163c1095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 --2- 172.21.15.45:0/2444494165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c1095d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1630009880 tx=0x7f163002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 -- 172.21.15.45:0/2444494165 shutdown_connections 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 --2- 172.21.15.45:0/2444494165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c1095d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 --2- 172.21.15.45:0/2444494165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f163c101c80 0x7f163c102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 -- 172.21.15.45:0/2444494165 >> 172.21.15.45:0/2444494165 conn(0x7f163c079bd0 msgr2=0x7f163c079fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 -- 172.21.15.45:0/2444494165 shutdown_connections 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.241+0000 7f164162b640 1 -- 172.21.15.45:0/2444494165 wait complete. 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f164162b640 1 Processor -- start 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f164162b640 1 -- start start 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f164162b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f163c101c80 0x7f163c1998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f164162b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:36.254 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f164162b640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f163c1108f0 con 0x7f163c1091c0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f164162b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f163c110770 con 0x7f163c101c80 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163a7fc640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c199e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:52910/0 (socket says 172.21.15.45:52910) 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163a7fc640 1 -- 172.21.15.45:0/3945024476 learned_addr learned my addr 172.21.15.45:0/3945024476 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163affd640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f163c101c80 0x7f163c1998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163a7fc640 1 -- 172.21.15.45:0/3945024476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f163c101c80 msgr2=0x7f163c1998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163a7fc640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f163c101c80 0x7f163c1998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.242+0000 7f163a7fc640 1 -- 172.21.15.45:0/3945024476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c19f100 con 0x7f163c1091c0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.243+0000 7f163a7fc640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c199e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f163002a9c0 tx=0x7f16300045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.243+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f163003f070 con 0x7f163c1091c0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.243+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c19f390 con 0x7f163c1091c0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.243+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16300049d0 con 0x7f163c1091c0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.243+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f1630035030 con 0x7f163c1091c0 2024-09-21T01:28:36.255 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.243+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c19f940 con 0x7f163c1091c0 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.244+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163c06b580 con 0x7f163c1091c0 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.245+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f1630042070 con 0x7f163c1091c0 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.245+0000 7f1623fff640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f162407af30 0x7f162407d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.245+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7138+0+0 (secure 0 0 0) 0x7f16300bef70 con 0x7f163c1091c0 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.245+0000 7f163affd640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f162407af30 0x7f162407d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.246+0000 7f163affd640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f162407af30 0x7f162407d440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f162c0058c0 tx=0x7f162c00a4c0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:36.256 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.250+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f163008a100 con 0x7f163c1091c0 2024-09-21T01:28:36.458 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.458+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f163c19a540 con 0x7f163c1091c0 2024-09-21T01:28:36.460 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.460+0000 7f1623fff640 1 -- 172.21.15.45:0/3945024476 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+21312 (secure 0 0 0) 0x7f163008efb0 con 0x7f163c1091c0 2024-09-21T01:28:36.460 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:36.460 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":62,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","created":"2024-09-21T01:20:38.644738+0000","modified":"2024-09-21T01:28:29.837269+0000","last_up_change":"2024-09-21T01:27:36.610150+0000","last_in_change":"2024-09-21T01:27:14.953689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:25:30.425195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T01:28:21.490666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T01:28:21.769309+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-21T01:28:25.828284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-21T01:28:26.813913+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20b68c0-eefc-40c0-b952-799c83b7f4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6803","nonce":3771776284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6805","nonce":3771776284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6809","nonce":3771776284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6807","nonce":3771776284}]},"public_addr":"172.21.15.45:6803/3771776284","cluster_addr":"172.21.15.45:6805/3771776284","heartbeat_back_addr":"172.21.15.45:6809/3771776284","heartbeat_front_addr":"172.21.15.45:6807/3771776284","state":["exists","up"]},{"osd":1,"uuid":"b10dd316-1566-4bff-933b-1b96b16a3378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6811","nonce":2203931892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6813","nonce":2203931892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6817","nonce":2203931892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6815","nonce":2203931892}]},"public_addr":"172.21.15.45:6811/2203931892","cluster_addr":"172.21.15.45:6813/2203931892","heartbeat_back_addr":"172.21.15.45:6817/2203931892","heartbeat_front_addr":"172.21.15.45:6815/2203931892","state":["exists","up"]},{"osd":2,"uuid":"80cb32e4-abb7-4cd6-b2af-6e254e3c592b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6819","nonce":2681427309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6821","nonce":2681427309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6825","nonce":2681427309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6823","nonce":2681427309}]},"public_addr":"172.21.15.45:6819/2681427309","cluster_addr":"172.21.15.45:6821/2681427309","heartbeat_back_addr":"172.21.15.45:6825/2681427309","heartbeat_front_addr":"172.21.15.45:6823/2681427309","state":["exists","up"]},{"osd":3,"uuid":"1aaec02f-d282-4505-a860-3d77532911df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6827","nonce":2480895441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6829","nonce":2480895441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6833","nonce":2480895441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6831","nonce":2480895441}]},"public_addr":"172.21.15.45:6827/2480895441","cluster_addr":"172.21.15.45:6829/2480895441","heartbeat_back_addr":"172.21.15.45:6833/2480895441","heartbeat_front_addr":"172.21.15.45:6831/2480895441","state":["exists","up"]},{"osd":4,"uuid":"85e89409-05b7-432d-9044-3b730f4f68b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6801","nonce":2331499915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6803","nonce":2331499915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6807","nonce":2331499915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6805","nonce":2331499915}]},"public_addr":"172.21.15.90:6801/2331499915","cluster_addr":"172.21.15.90:6803/2331499915","heartbeat_back_addr":"172.21.15.90:6807/2331499915","heartbeat_front_addr":"172.21.15.90:6805/2331499915","state":["exists","up"]},{"osd":5,"uuid":"0122a5d6-f32b-4851-851e-aa054f411b75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6809","nonce":1613366233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6811","nonce":1613366233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6815","nonce":1613366233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6813","nonce":1613366233}]},"public_addr":"172.21.15.90:6809/1613366233","cluster_addr":"172.21.15.90:6811/1613366233","heartbeat_back_addr":"172.21.15.90:6815/1613366233","heartbeat_front_addr":"172.21.15.90:6813/1613366233","state":["exists","up"]},{"osd":6,"uuid":"5170fdbb-6229-40f8-b415-de7d628a0490","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6817","nonce":2608695333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6819","nonce":2608695333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6823","nonce":2608695333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6821","nonce":2608695333}]},"public_addr":"172.21.15.90:6817/2608695333","cluster_addr":"172.21.15.90:6819/2608695333","heartbeat_back_addr":"172.21.15.90:6823/2608695333","heartbeat_front_addr":"172.21.15.90:6821/2608695333","state":["exists","up"]},{"osd":7,"uuid":"3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6825","nonce":2703954098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6827","nonce":2703954098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6831","nonce":2703954098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6829","nonce":2703954098}]},"public_addr":"172.21.15.90:6825/2703954098","cluster_addr":"172.21.15.90:6827/2703954098","heartbeat_back_addr":"172.21.15.90:6831/2703954098","heartbeat_front_addr":"172.21.15.90:6829/2703954098","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:24:32.659134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:00.007423+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:27.198145+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:54.790013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:18.251812+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:43.822784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:08.545224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:34.701933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.45:0/1147989706":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1797066051":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3851449798":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/3707884113":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3194963614":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6800/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1334988077":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6800/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/1726148947":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/308996383":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6801/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:6800/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:0/1437053407":"2024-09-22T01:21:45.172023+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:28:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f162407af30 msgr2=0x7f162407d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:36.462 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f162407af30 0x7f162407d440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f162c0058c0 tx=0x7f162c00a4c0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 msgr2=0x7f163c199e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c199e60 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f163002a9c0 tx=0x7f16300045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 shutdown_connections 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f162407af30 0x7f162407d440 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f163c1091c0 0x7f163c199e60 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 --2- 172.21.15.45:0/3945024476 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f163c101c80 0x7f163c1998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 >> 172.21.15.45:0/3945024476 conn(0x7f163c079bd0 msgr2=0x7f163c1072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.462+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 shutdown_connections 2024-09-21T01:28:36.463 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:36.463+0000 7f164162b640 1 -- 172.21.15.45:0/3945024476 wait complete. 2024-09-21T01:28:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:36 smithi045 ceph-mon[28361]: pgmap v222: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s wr, 10 op/s 2024-09-21T01:28:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:36 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3945024476' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:28:37.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:36 smithi090 ceph-mon[35142]: pgmap v222: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s wr, 10 op/s 2024-09-21T01:28:37.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:36 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3945024476' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:28:37.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 5, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-21T01:28:22.773074+0000', 'modified': '2024-09-21T01:28:27.802694+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14620}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14620': {'gid': 14620, 'name': 'foo.smithi045.qpbfpk', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.45:6835/4020685920', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6834', 'nonce': 4020685920}, {'type': 'v1', 'addr': '172.21.15.45:6835', 'nonce': 4020685920}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'foo', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 14620, 'qdb_cluster': [14620]} 2024-09-21T01:28:37.898 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-21T01:28:37.898 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell mds.foo.smithi045.qpbfpk status 2024-09-21T01:28:38.336 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:38.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:38.582 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:38.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:38.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 -- 172.21.15.45:0/981350484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 msgr2=0x7f6d3c1084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 --2- 172.21.15.45:0/981350484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c1084a0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009880 tx=0x7f6d2c02a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 -- 172.21.15.45:0/981350484 shutdown_connections 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 --2- 172.21.15.45:0/981350484 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c1084a0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 --2- 172.21.15.45:0/981350484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c102750 0x7f6d3c102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 -- 172.21.15.45:0/981350484 >> 172.21.15.45:0/981350484 conn(0x7f6d3c0fe1a0 msgr2=0x7f6d3c1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 -- 172.21.15.45:0/981350484 shutdown_connections 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.023+0000 7f6d435dd640 1 -- 172.21.15.45:0/981350484 wait complete. 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d435dd640 1 Processor -- start 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d435dd640 1 -- start start 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d435dd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c102750 0x7f6d3c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.025 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d435dd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c19ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d435dd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c10f7c0 con 0x7f6d3c108090 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d435dd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c10f640 con 0x7f6d3c102750 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d40b78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c19ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d40b78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c19ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42118/0 (socket says 172.21.15.45:42118) 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d40b78640 1 -- 172.21.15.45:0/4066188301 learned_addr learned my addr 172.21.15.45:0/4066188301 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.024+0000 7f6d41379640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c102750 0x7f6d3c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d40b78640 1 -- 172.21.15.45:0/4066188301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c102750 msgr2=0x7f6d3c19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d40b78640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c102750 0x7f6d3c19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d40b78640 1 -- 172.21.15.45:0/4066188301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3c19b0c0 con 0x7f6d3c108090 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d40b78640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c19ab80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c02a9c0 tx=0x7f6d2c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d327fc640 1 -- 172.21.15.45:0/4066188301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d2c03f070 con 0x7f6d3c108090 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d327fc640 1 -- 172.21.15.45:0/4066188301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d2c004950 con 0x7f6d3c108090 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d3c19b350 con 0x7f6d3c108090 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d327fc640 1 -- 172.21.15.45:0/4066188301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d2c035030 con 0x7f6d3c108090 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.025+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d3c1a3880 con 0x7f6d3c108090 2024-09-21T01:28:39.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.027+0000 7f6d327fc640 1 -- 172.21.15.45:0/4066188301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f6d2c042070 con 0x7f6d3c108090 2024-09-21T01:28:39.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.028+0000 7f6d327fc640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6d2807af30 0x7f6d2807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.028+0000 7f6d49b51740 1 Processor -- start 2024-09-21T01:28:39.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.028+0000 7f6d49b51740 1 -- start start 2024-09-21T01:28:39.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.028+0000 7f6d327fc640 1 -- 172.21.15.45:0/4066188301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7138+0+0 (secure 0 0 0) 0x7f6d2c0bef10 con 0x7f6d3c108090 2024-09-21T01:28:39.028 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.028+0000 7f6d41379640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6d2807af30 0x7f6d2807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.029 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23b1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d49b51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561cf2329d00 con 0x561cf200c0d0 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d49b51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561cf2319130 con 0x561cf1fe3210 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23b1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23b1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42120/0 (socket says 172.21.15.45:42120) 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 -- 172.21.15.45:0/1653771458 learned_addr learned my addr 172.21.15.45:0/1653771458 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d40b78640 1 --2- 172.21.15.45:0/1653771458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6d2807af30 0x7f6d2807d440 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c06b710 tx=0x7f6d38003d40 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 -- 172.21.15.45:0/1653771458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 msgr2=0x561cf23b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 --2- 172.21.15.45:0/1653771458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 -- 172.21.15.45:0/1653771458 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cf20776b0 con 0x561cf200c0d0 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d41379640 1 --2- 172.21.15.45:0/1653771458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23b1e80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6d380134e0 tx=0x7f6d3801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223db8c6fb65e4f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:39.030 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.029+0000 7f6d277fe640 1 -- 172.21.15.45:0/1653771458 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d3801e420 con 0x561cf200c0d0 2024-09-21T01:28:39.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d277fe640 1 -- 172.21.15.45:0/1653771458 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d3801ea50 con 0x561cf200c0d0 2024-09-21T01:28:39.031 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d277fe640 1 -- 172.21.15.45:0/1653771458 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d38011790 con 0x561cf200c0d0 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d49b51740 1 -- 172.21.15.45:0/1653771458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 msgr2=0x561cf23b1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d49b51740 1 --2- 172.21.15.45:0/1653771458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23b1e80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6d380134e0 tx=0x7f6d3801b000 comp rx=0 tx=0).stop 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d49b51740 1 -- 172.21.15.45:0/1653771458 shutdown_connections 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d49b51740 1 --2- 172.21.15.45:0/1653771458 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23b23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d49b51740 1 --2- 172.21.15.45:0/1653771458 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23b1e80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.030+0000 7f6d49b51740 1 -- 172.21.15.45:0/1653771458 >> 172.21.15.45:0/1653771458 conn(0x561cf20004c0 msgr2=0x561cf23afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.031+0000 7f6d49b51740 1 -- 172.21.15.45:0/1653771458 shutdown_connections 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.031+0000 7f6d49b51740 1 -- 172.21.15.45:0/1653771458 wait complete. 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.031+0000 7f6d49b51740 1 -- start start 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.031+0000 7f6d49b51740 1 Processor -- start 2024-09-21T01:28:39.032 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23bdb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23be130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d49b51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561cf22b4100 con 0x561cf200c0d0 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d49b51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561cf231cf40 con 0x561cf1fe3210 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d49b51740 10 client.0 _ll_register_callbacks cb 0x561cf2313430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d41379640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23bdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d40b78640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23be130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d41379640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23bdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.45:37340/0 (socket says 172.21.15.45:37340) 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d41379640 1 -- 172.21.15.45:0/3287827353 learned_addr learned my addr 172.21.15.45:0/3287827353 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d40b78640 1 -- 172.21.15.45:0/3287827353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 msgr2=0x561cf23bdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d40b78640 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23bdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.033 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d40b78640 1 -- 172.21.15.45:0/3287827353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cf22e1af0 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d40b78640 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23be130 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c082840 tx=0x7f6d2c08a0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29db0dd9b168fb9a server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3287827353 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d2c049040 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3287827353 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(62..62 src has 1..62) ==== 7138+0+0 (secure 0 0 0) 0x7f6d2c081840 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.032+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d3801ea50 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3287827353 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f6d2c084830 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 10 client.14644 fetch_fsmap learned FSMap version 9 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=3+}) -- 0x561cf23c9910 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3287827353 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d3801ea50 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3287827353 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 9) ==== 2968+0+0 (secure 0 0 0) 0x7f6d2c09f070 con 0x561cf200c0d0 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 10 client.14644 fetch_fsmap finished waiting for FSMap version 9 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 10 client.14644 resolve_mds: resolved name 'foo.smithi045.qpbfpk' to daemon mds.foo.smithi045.qpbfpk 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 20 client.14644 populate_metadata read hostname 'smithi045' 2024-09-21T01:28:39.036 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1feb3f0 0x561cf23cd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 4 client.14644 mds_command: new command op to 14620 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 --> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561cf232bc50 con 0x561cf1feb3f0 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.033+0000 7f6d41379640 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1feb3f0 0x561cf23cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.034+0000 7f6d24ff9640 10 client.14644 ms_handle_connect on v2:172.21.15.45:6834/4020685920 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.034+0000 7f6d41379640 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1feb3f0 0x561cf23cd790 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6e31ba814d0c5f1 server_cookie=b4bfcd61e58d4aec in_seq=0 out_seq=0 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3287827353 <== mds.0 v2:172.21.15.45:6834/4020685920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x561cf232bc50 con 0x561cf1feb3f0 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d24ff9640 10 client.14644 handle_command_reply: tid=0 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 client.14644 shutdown 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 msgr2=0x561cf23be130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23be130 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c082840 tx=0x7f6d2c08a0c0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 shutdown_connections 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1feb3f0 0x561cf23cd790 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf200c0d0 0x561cf23be130 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.037 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3287827353 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf1fe3210 0x561cf23bdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 >> 172.21.15.45:0/3287827353 conn(0x561cf20004c0 msgr2=0x561cf23b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:39.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d24ff9640 0 client.14644 ms_handle_reset on v2:172.21.15.45:6834/4020685920 2024-09-21T01:28:39.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 shutdown_connections 2024-09-21T01:28:39.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 1 -- 172.21.15.45:0/3287827353 wait complete. 2024-09-21T01:28:39.038 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.035+0000 7f6d49b51740 20 client.14644 trim_cache size 0 max 16384 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.055+0000 7f6d49b51740 1 Processor -- start 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.055+0000 7f6d49b51740 1 -- start start 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d49b51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c1aa640 con 0x561cf1fe3210 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d49b51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f6d3c1ab640 con 0x561cf200c0d0 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23af0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42140/0 (socket says 172.21.15.45:42140) 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 -- 172.21.15.45:0/2353579789 learned_addr learned my addr 172.21.15.45:0/2353579789 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d40b78640 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.057 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 -- 172.21.15.45:0/2353579789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 msgr2=0x561cf23c26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23c26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 -- 172.21.15.45:0/2353579789 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561cf23bc8b0 con 0x561cf1fe3210 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d40b78640 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.056+0000 7f6d41379640 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23af0d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6d38013fd0 tx=0x7f6d3801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17278832ec1fd125 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.057+0000 7f6d26ffd640 1 -- 172.21.15.45:0/2353579789 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d3803c070 con 0x561cf1fe3210 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.057+0000 7f6d26ffd640 1 -- 172.21.15.45:0/2353579789 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d3801e420 con 0x561cf1fe3210 2024-09-21T01:28:39.058 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.057+0000 7f6d26ffd640 1 -- 172.21.15.45:0/2353579789 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d38011870 con 0x561cf1fe3210 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.057+0000 7f6d49b51740 1 -- 172.21.15.45:0/2353579789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 msgr2=0x561cf23af0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.057+0000 7f6d49b51740 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23af0d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6d38013fd0 tx=0x7f6d3801c000 comp rx=0 tx=0).stop 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 -- 172.21.15.45:0/2353579789 shutdown_connections 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23c26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 --2- 172.21.15.45:0/2353579789 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23af0d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 -- 172.21.15.45:0/2353579789 >> 172.21.15.45:0/2353579789 conn(0x561cf20004c0 msgr2=0x561cf23c1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 -- 172.21.15.45:0/2353579789 shutdown_connections 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 -- 172.21.15.45:0/2353579789 wait complete. 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 -- start start 2024-09-21T01:28:39.060 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.058+0000 7f6d49b51740 1 Processor -- start 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23d0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d49b51740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d49b51740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x561cf23c9f90 con 0x561cf1fe3210 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d49b51740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x561cf217a760 con 0x561cf200c0d0 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d49b51740 10 client.0 _ll_register_callbacks cb 0x7f6d2c0850e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23d0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23d0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42156/0 (socket says 172.21.15.45:42156) 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 -- 172.21.15.45:0/3151227767 learned_addr learned my addr 172.21.15.45:0/3151227767 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d40b78640 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 -- 172.21.15.45:0/3151227767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 msgr2=0x561cf23d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:39.061 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23d11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 -- 172.21.15.45:0/3151227767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561cf23cb870 con 0x561cf1fe3210 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d40b78640 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23d11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d41379640 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23d0c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6d38004480 tx=0x7f6d380047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59252672476b16a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.059+0000 7f6d24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3151227767 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f6d3803c040 con 0x561cf1fe3210 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3151227767 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(62..62 src has 1..62) ==== 7138+0+0 (secure 0 0 0) 0x7f6d3802b040 con 0x561cf1fe3210 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d3801e420 con 0x561cf1fe3210 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3151227767 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f6d3804e050 con 0x561cf1fe3210 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 10 client.14648 fetch_fsmap learned FSMap version 9 2024-09-21T01:28:39.062 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=3+}) -- 0x561cf24380f0 con 0x561cf1fe3210 2024-09-21T01:28:39.063 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3151227767 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f6d3801e420 con 0x561cf1fe3210 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3151227767 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 9) ==== 2968+0+0 (secure 0 0 0) 0x7f6d38051020 con 0x561cf1fe3210 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 10 client.14648 fetch_fsmap finished waiting for FSMap version 9 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 10 client.14648 resolve_mds: resolved name 'foo.smithi045.qpbfpk' to daemon mds.foo.smithi045.qpbfpk 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 20 client.14648 populate_metadata read hostname 'smithi045' 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1fcaa50 0x561cf243b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 4 client.14648 mds_command: new command op to 14620 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.060+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 --> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] -- command(tid 0: {"prefix": "status"}) -- 0x561cf2327700 con 0x561cf1fcaa50 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.061+0000 7f6d40b78640 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1fcaa50 0x561cf243b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.061+0000 7f6d24ff9640 10 client.14648 ms_handle_connect on v2:172.21.15.45:6834/4020685920 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.061+0000 7f6d40b78640 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1fcaa50 0x561cf243b050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf380c85749ec06c server_cookie=3112dbcdd5eea92f in_seq=0 out_seq=0 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.061+0000 7f6d24ff9640 1 -- 172.21.15.45:0/3151227767 <== mds.0 v2:172.21.15.45:6834/4020685920 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x561cf2327700 con 0x561cf1fcaa50 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.061+0000 7f6d24ff9640 10 client.14648 handle_command_reply: tid=0 2024-09-21T01:28:39.064 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.061+0000 7f6d49b51740 1 client.14648 shutdown 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 msgr2=0x561cf23d0c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23d0c10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6d38004480 tx=0x7f6d380047e0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 shutdown_connections 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x561cf200c0d0 0x561cf23d11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:6834/4020685920,v1:172.21.15.45:6835/4020685920] conn(0x561cf1fcaa50 0x561cf243b050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 --2- 172.21.15.45:0/3151227767 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x561cf1fe3210 0x561cf23d0c10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 >> 172.21.15.45:0/3151227767 conn(0x561cf20004c0 msgr2=0x561cf23b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d24ff9640 0 client.14648 ms_handle_reset on v2:172.21.15.45:6834/4020685920 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 shutdown_connections 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 1 -- 172.21.15.45:0/3151227767 wait complete. 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.062+0000 7f6d49b51740 20 client.14648 trim_cache size 0 max 16384 2024-09-21T01:28:39.065 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9b2fd594-77b7-11ef-baf5-efdc52797490", 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 14620, 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "foo", 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 12.255034348000001, 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 9, 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 62, 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 62, 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 12.959581133 2024-09-21T01:28:39.066 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6d2807af30 msgr2=0x7f6d2807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6d2807af30 0x7f6d2807d440 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c06b710 tx=0x7f6d38003d40 comp rx=0 tx=0).stop 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 msgr2=0x7f6d3c19ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c19ab80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c02a9c0 tx=0x7f6d2c0045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 shutdown_connections 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f6d2807af30 0x7f6d2807d440 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.067 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6d3c108090 0x7f6d3c19ab80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 --2- 172.21.15.45:0/4066188301 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d3c102750 0x7f6d3c19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:39.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 >> 172.21.15.45:0/4066188301 conn(0x7f6d3c0fe1a0 msgr2=0x7f6d3c1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:39.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 shutdown_connections 2024-09-21T01:28:39.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:39.063+0000 7f6d435dd640 1 -- 172.21.15.45:0/4066188301 wait complete. 2024-09-21T01:28:39.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:39 smithi045 ceph-mon[28361]: pgmap v223: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s wr, 9 op/s 2024-09-21T01:28:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:39 smithi090 ceph-mon[35142]: pgmap v223: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s wr, 9 op/s 2024-09-21T01:28:39.828 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:40.380 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 -- 172.21.15.45:0/1501947707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0109190 msgr2=0x7f9ba01095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 --2- 172.21.15.45:0/1501947707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0109190 0x7f9ba01095a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009880 tx=0x7f9b8802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 -- 172.21.15.45:0/1501947707 shutdown_connections 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 --2- 172.21.15.45:0/1501947707 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0109190 0x7f9ba01095a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 --2- 172.21.15.45:0/1501947707 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba0103850 0x7f9ba0103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 -- 172.21.15.45:0/1501947707 >> 172.21.15.45:0/1501947707 conn(0x7f9ba00ff2a0 msgr2=0x7f9ba01016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 -- 172.21.15.45:0/1501947707 shutdown_connections 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.841+0000 7f9ba5740640 1 -- 172.21.15.45:0/1501947707 wait complete. 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9ba5740640 1 Processor -- start 2024-09-21T01:28:40.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9ba5740640 1 -- start start 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9ba5740640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 0x7f9ba019ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9ba5740640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba0109190 0x7f9ba019c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9ba5740640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba01108c0 con 0x7f9ba0103850 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9ba5740640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba0110740 con 0x7f9ba0109190 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b9effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 0x7f9ba019ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b9effd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 0x7f9ba019ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42200/0 (socket says 172.21.15.45:42200) 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b9effd640 1 -- 172.21.15.45:0/3819491029 learned_addr learned my addr 172.21.15.45:0/3819491029 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b97fff640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba0109190 0x7f9ba019c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b9effd640 1 -- 172.21.15.45:0/3819491029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba0109190 msgr2=0x7f9ba019c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b9effd640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba0109190 0x7f9ba019c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.842+0000 7f9b9effd640 1 -- 172.21.15.45:0/3819491029 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba01a12d0 con 0x7f9ba0103850 2024-09-21T01:28:40.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.843+0000 7f9b9effd640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 0x7f9ba019ba90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c8a0 tx=0x7f9b9000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.843+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9b900079b0 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.843+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9b9000f450 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.843+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba01a15c0 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.843+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9b90018630 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.843+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba01a1ba0 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.844+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ba0104ad0 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.845+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f9b90007b50 con 0x7f9ba0103850 2024-09-21T01:28:40.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.845+0000 7f9b9cff9640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f9b8c07af30 0x7f9b8c07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.846+0000 7f9b97fff640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f9b8c07af30 0x7f9b8c07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.846+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7f9b9009e740 con 0x7f9ba0103850 2024-09-21T01:28:40.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.846+0000 7f9b97fff640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f9b8c07af30 0x7f9b8c07d440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009230 tx=0x7f9b880058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:40.850 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:40.850+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9b90014030 con 0x7f9ba0103850 2024-09-21T01:28:41.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.079+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ba019c740 con 0x7f9ba0103850 2024-09-21T01:28:41.079 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.079+0000 7f9b9cff9640 1 -- 172.21.15.45:0/3819491029 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+4919 (secure 0 0 0) 0x7f9b900697e0 con 0x7f9ba0103850 2024-09-21T01:28:41.079 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:41.080 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":11,"btime":"2024-09-21T01:28:40:410001+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:22.773074+0000","modified":"2024-09-21T01:28:30.825886+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14620},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14620":{"gid":14620,"name":"foo.smithi045.qpbfpk","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/4020685920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4020685920},{"type":"v1","addr":"172.21.15.45:6835","nonce":4020685920}]},"join_fscid":1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14620,"qdb_cluster":[14620]},"id":1},{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:27.820077+0000","modified":"2024-09-21T01:28:40.409996+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"bar.smithi045.tspicn","rank":0,"incarnation":11,"state":"up:replay","state_seq":1,"addr":"172.21.15.45:6837/1900773137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1900773137},{"type":"v1","addr":"172.21.15.45:6837","nonce":1900773137}]},"join_fscid":-1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-09-21T01:28:41.080 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 11 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f9b8c07af30 msgr2=0x7f9b8c07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f9b8c07af30 0x7f9b8c07d440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9b88009230 tx=0x7f9b880058d0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 msgr2=0x7f9ba019ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 0x7f9ba019ba90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c8a0 tx=0x7f9b9000cd60 comp rx=0 tx=0).stop 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 shutdown_connections 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f9b8c07af30 0x7f9b8c07d440 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.082 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9ba0109190 0x7f9ba019c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 --2- 172.21.15.45:0/3819491029 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ba0103850 0x7f9ba019ba90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:41.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 >> 172.21.15.45:0/3819491029 conn(0x7f9ba00ff2a0 msgr2=0x7f9ba00ff6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:41.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 shutdown_connections 2024-09-21T01:28:41.083 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:41.082+0000 7f9ba5740640 1 -- 172.21.15.45:0/3819491029 wait complete. 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: pgmap v224: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s wr, 7 op/s 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Cluster is now healthy 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:boot 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Dropping low affinity active daemon mds.foo.smithi090.savegz in favor of higher affinity standby. 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Replacing daemon mds.foo.smithi090.savegz as rank 0 with standby daemon mds.bar.smithi045.tspicn 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T01:28:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "bar.smithi045.tspicn"} : dispatch 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi045.tspicn=up:replay,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.qzpqdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.qzpqdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: Deploying daemon mds.bar.smithi090.qzpqdi on smithi090 2024-09-21T01:28:41.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:41 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3819491029' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: pgmap v224: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s wr, 7 op/s 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Cluster is now healthy 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:boot 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Dropping low affinity active daemon mds.foo.smithi090.savegz in favor of higher affinity standby. 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Replacing daemon mds.foo.smithi090.savegz as rank 0 with standby daemon mds.bar.smithi045.tspicn 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=foo.smithi090.savegz=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:41.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "bar.smithi045.tspicn"} : dispatch 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi045.tspicn=up:replay,foo:0=foo.smithi045.qpbfpk=up:active} 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.qzpqdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.qzpqdi", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: Deploying daemon mds.bar.smithi090.qzpqdi on smithi090 2024-09-21T01:28:41.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:41 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3819491029' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:41.831 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-21T01:28:42.095 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:42.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:42 smithi090 ceph-mon[35142]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-21T01:28:42.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:42 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:reconnect 2024-09-21T01:28:42.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:42 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] up:boot 2024-09-21T01:28:42.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:42 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi045.tspicn=up:reconnect,foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:42.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:42 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "foo.smithi090.savegz"} : dispatch 2024-09-21T01:28:42.512 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:42 smithi045 ceph-mon[28361]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-21T01:28:42.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:42 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:reconnect 2024-09-21T01:28:42.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:42 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] up:boot 2024-09-21T01:28:42.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:42 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi045.tspicn=up:reconnect,foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:42.513 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:42 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "foo.smithi090.savegz"} : dispatch 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 -- 172.21.15.45:0/1588505747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 msgr2=0x7f1708103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 --2- 172.21.15.45:0/1588505747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708103c60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009880 tx=0x7f16f002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 -- 172.21.15.45:0/1588505747 shutdown_connections 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 --2- 172.21.15.45:0/1588505747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1708109190 0x7f17081095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 --2- 172.21.15.45:0/1588505747 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708103c60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 -- 172.21.15.45:0/1588505747 >> 172.21.15.45:0/1588505747 conn(0x7f17080ff2a0 msgr2=0x7f17081016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.512+0000 7f170eb7c640 1 -- 172.21.15.45:0/1588505747 shutdown_connections 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 -- 172.21.15.45:0/1588505747 wait complete. 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 Processor -- start 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 -- start start 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708075710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:42.520 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1708109190 0x7f1708075c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f1708110920 con 0x7f1708103850 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.513+0000 7f170eb7c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f17081107a0 con 0x7f1708109190 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708075710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708075710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42208/0 (socket says 172.21.15.45:42208) 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 -- 172.21.15.45:0/3397543766 learned_addr learned my addr 172.21.15.45:0/3397543766 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f1707fff640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1708109190 0x7f1708075c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 -- 172.21.15.45:0/3397543766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1708109190 msgr2=0x7f1708075c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1708109190 0x7f1708075c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 -- 172.21.15.45:0/3397543766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17080775f0 con 0x7f1708103850 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170c918640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708075710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f16f002a9c0 tx=0x7f16f00045c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f16f003f070 con 0x7f1708103850 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f16f0004990 con 0x7f1708103850 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f16f0035030 con 0x7f1708103850 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.514+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1708076250 con 0x7f1708103850 2024-09-21T01:28:42.521 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.515+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17081a5bf0 con 0x7f1708103850 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.516+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1708104ad0 con 0x7f1708103850 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.516+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f16f0042070 con 0x7f1708103850 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.517+0000 7f1705ffb640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f16f407af80 0x7f16f407d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.517+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7f16f00bed30 con 0x7f1708103850 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.517+0000 7f1707fff640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f16f407af80 0x7f16f407d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.517+0000 7f1707fff640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f16f407af80 0x7f16f407d490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1708108c90 tx=0x7f16f8009500 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:42.522 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.522+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f16f0089ef0 con 0x7f1708103850 2024-09-21T01:28:42.711 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.711+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f170810b930 con 0x7f1708103850 2024-09-21T01:28:42.712 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.712+0000 7f1705ffb640 1 -- 172.21.15.45:0/3397543766 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) ==== 74+0+21442 (secure 0 0 0) 0x7f16f008eda0 con 0x7f1708103850 2024-09-21T01:28:42.712 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:42.712 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":63,"fsid":"9b2fd594-77b7-11ef-baf5-efdc52797490","created":"2024-09-21T01:20:38.644738+0000","modified":"2024-09-21T01:28:40.409814+0000","last_up_change":"2024-09-21T01:27:36.610150+0000","last_in_change":"2024-09-21T01:27:14.953689+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T01:25:30.425195+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T01:28:21.490666+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T01:28:21.769309+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-21T01:28:25.828284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-21T01:28:26.813913+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e20b68c0-eefc-40c0-b952-799c83b7f4d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6802","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6803","nonce":3771776284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6804","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6805","nonce":3771776284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6808","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6809","nonce":3771776284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6806","nonce":3771776284},{"type":"v1","addr":"172.21.15.45:6807","nonce":3771776284}]},"public_addr":"172.21.15.45:6803/3771776284","cluster_addr":"172.21.15.45:6805/3771776284","heartbeat_back_addr":"172.21.15.45:6809/3771776284","heartbeat_front_addr":"172.21.15.45:6807/3771776284","state":["exists","up"]},{"osd":1,"uuid":"b10dd316-1566-4bff-933b-1b96b16a3378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6810","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6811","nonce":2203931892}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6812","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6813","nonce":2203931892}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6816","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6817","nonce":2203931892}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6814","nonce":2203931892},{"type":"v1","addr":"172.21.15.45:6815","nonce":2203931892}]},"public_addr":"172.21.15.45:6811/2203931892","cluster_addr":"172.21.15.45:6813/2203931892","heartbeat_back_addr":"172.21.15.45:6817/2203931892","heartbeat_front_addr":"172.21.15.45:6815/2203931892","state":["exists","up"]},{"osd":2,"uuid":"80cb32e4-abb7-4cd6-b2af-6e254e3c592b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6818","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6819","nonce":2681427309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6820","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6821","nonce":2681427309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6824","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6825","nonce":2681427309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6822","nonce":2681427309},{"type":"v1","addr":"172.21.15.45:6823","nonce":2681427309}]},"public_addr":"172.21.15.45:6819/2681427309","cluster_addr":"172.21.15.45:6821/2681427309","heartbeat_back_addr":"172.21.15.45:6825/2681427309","heartbeat_front_addr":"172.21.15.45:6823/2681427309","state":["exists","up"]},{"osd":3,"uuid":"1aaec02f-d282-4505-a860-3d77532911df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6826","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6827","nonce":2480895441}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6828","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6829","nonce":2480895441}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6832","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6833","nonce":2480895441}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6830","nonce":2480895441},{"type":"v1","addr":"172.21.15.45:6831","nonce":2480895441}]},"public_addr":"172.21.15.45:6827/2480895441","cluster_addr":"172.21.15.45:6829/2480895441","heartbeat_back_addr":"172.21.15.45:6833/2480895441","heartbeat_front_addr":"172.21.15.45:6831/2480895441","state":["exists","up"]},{"osd":4,"uuid":"85e89409-05b7-432d-9044-3b730f4f68b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6801","nonce":2331499915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6803","nonce":2331499915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6807","nonce":2331499915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":2331499915},{"type":"v1","addr":"172.21.15.90:6805","nonce":2331499915}]},"public_addr":"172.21.15.90:6801/2331499915","cluster_addr":"172.21.15.90:6803/2331499915","heartbeat_back_addr":"172.21.15.90:6807/2331499915","heartbeat_front_addr":"172.21.15.90:6805/2331499915","state":["exists","up"]},{"osd":5,"uuid":"0122a5d6-f32b-4851-851e-aa054f411b75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6809","nonce":1613366233}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6811","nonce":1613366233}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6815","nonce":1613366233}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":1613366233},{"type":"v1","addr":"172.21.15.90:6813","nonce":1613366233}]},"public_addr":"172.21.15.90:6809/1613366233","cluster_addr":"172.21.15.90:6811/1613366233","heartbeat_back_addr":"172.21.15.90:6815/1613366233","heartbeat_front_addr":"172.21.15.90:6813/1613366233","state":["exists","up"]},{"osd":6,"uuid":"5170fdbb-6229-40f8-b415-de7d628a0490","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6817","nonce":2608695333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6819","nonce":2608695333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6823","nonce":2608695333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":2608695333},{"type":"v1","addr":"172.21.15.90:6821","nonce":2608695333}]},"public_addr":"172.21.15.90:6817/2608695333","cluster_addr":"172.21.15.90:6819/2608695333","heartbeat_back_addr":"172.21.15.90:6823/2608695333","heartbeat_front_addr":"172.21.15.90:6821/2608695333","state":["exists","up"]},{"osd":7,"uuid":"3d454b74-a12f-4c7f-bf9c-ff328a7cc0a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6825","nonce":2703954098}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6827","nonce":2703954098}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6831","nonce":2703954098}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":2703954098},{"type":"v1","addr":"172.21.15.90:6829","nonce":2703954098}]},"public_addr":"172.21.15.90:6825/2703954098","cluster_addr":"172.21.15.90:6827/2703954098","heartbeat_back_addr":"172.21.15.90:6831/2703954098","heartbeat_front_addr":"172.21.15.90:6829/2703954098","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:24:32.659134+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:00.007423+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:27.198145+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:25:54.790013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:18.251812+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:26:43.822784+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:08.545224+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T01:27:34.701933+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:6833/3876994677":"2024-09-22T01:28:40.409775+0000","172.21.15.45:0/1147989706":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/1797066051":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3851449798":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.45:0/3707884113":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/3194963614":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6800/2275235431":"2024-09-22T01:22:51.944078+0000","172.21.15.90:6832/3876994677":"2024-09-22T01:28:40.409775+0000","172.21.15.45:0/1334988077":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6800/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/1726148947":"2024-09-22T01:21:11.091918+0000","172.21.15.45:0/308996383":"2024-09-22T01:22:51.944078+0000","172.21.15.45:6801/3815412594":"2024-09-22T01:21:11.091918+0000","172.21.15.45:6800/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:6801/2568383000":"2024-09-22T01:21:45.172023+0000","172.21.15.45:0/1437053407":"2024-09-22T01:21:45.172023+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.714+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f16f407af80 msgr2=0x7f16f407d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.714+0000 7f170eb7c640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f16f407af80 0x7f16f407d490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1708108c90 tx=0x7f16f8009500 comp rx=0 tx=0).stop 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.714+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 msgr2=0x7f1708075710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.714+0000 7f170eb7c640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708075710 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f16f002a9c0 tx=0x7f16f00045c0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 shutdown_connections 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f16f407af80 0x7f16f407d490 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1708109190 0x7f1708075c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 --2- 172.21.15.45:0/3397543766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1708103850 0x7f1708075710 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 >> 172.21.15.45:0/3397543766 conn(0x7f17080ff2a0 msgr2=0x7f17081016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:42.715 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 shutdown_connections 2024-09-21T01:28:42.716 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:42.715+0000 7f170eb7c640 1 -- 172.21.15.45:0/3397543766 wait complete. 2024-09-21T01:28:43.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-21T01:28:27.820077+0000', 'modified': '2024-09-21T01:28:29.837023+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24367}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24367': {'gid': 24367, 'name': 'foo.smithi090.savegz', 'rank': 0, 'incarnation': 7, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.90:6833/3876994677', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6832', 'nonce': 3876994677}, {'type': 'v1', 'addr': '172.21.15.90:6833', 'nonce': 3876994677}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'bar', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24367, 'qdb_cluster': [24367]} 2024-09-21T01:28:43.462 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-21T01:28:43.462 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell mds.foo.smithi090.savegz status 2024-09-21T01:28:43.684 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: pgmap v226: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 409 B/s wr, 3 op/s 2024-09-21T01:28:43.684 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.684 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:rejoin 2024-09-21T01:28:43.684 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi045.tspicn=up:rejoin,foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:43.684 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: daemon mds.bar.smithi045.tspicn is now active in filesystem bar as rank 0 2024-09-21T01:28:43.684 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/3397543766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:28:43.685 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.685 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.685 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.685 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.685 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.685 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:43 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:43.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: pgmap v226: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 409 B/s wr, 3 op/s 2024-09-21T01:28:43.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:rejoin 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi045.tspicn=up:rejoin,foo:0=foo.smithi045.qpbfpk=up:active} 1 up:standby 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: daemon mds.bar.smithi045.tspicn is now active in filesystem bar as rank 0 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/3397543766' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:43.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:43 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:44.377 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:44.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:44 smithi090 ceph-mon[35142]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T01:28:44.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:44 smithi090 ceph-mon[35142]: Cluster is now healthy 2024-09-21T01:28:44.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:44 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:active 2024-09-21T01:28:44.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:44 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.90:6834/764561033,v1:172.21.15.90:6835/764561033] up:boot 2024-09-21T01:28:44.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:44 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=bar.smithi045.tspicn=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2 up:standby 2024-09-21T01:28:44.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:44 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "bar.smithi090.qzpqdi"} : dispatch 2024-09-21T01:28:44.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:44 smithi045 ceph-mon[28361]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-21T01:28:44.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:44 smithi045 ceph-mon[28361]: Cluster is now healthy 2024-09-21T01:28:44.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:44 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:active 2024-09-21T01:28:44.731 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:44 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.90:6834/764561033,v1:172.21.15.90:6835/764561033] up:boot 2024-09-21T01:28:44.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:44 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=bar.smithi045.tspicn=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2 up:standby 2024-09-21T01:28:44.732 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:44 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "mds metadata", "who": "bar.smithi090.qzpqdi"} : dispatch 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 -- 172.21.15.45:0/1203824522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 msgr2=0x7f87a8104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 --2- 172.21.15.45:0/1203824522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a8104ab0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8798009880 tx=0x7f879802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 -- 172.21.15.45:0/1203824522 shutdown_connections 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 --2- 172.21.15.45:0/1203824522 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a8104ff0 0x7f87a810d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 --2- 172.21.15.45:0/1203824522 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a8104ab0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 -- 172.21.15.45:0/1203824522 >> 172.21.15.45:0/1203824522 conn(0x7f87a80fe1a0 msgr2=0x7f87a81005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 -- 172.21.15.45:0/1203824522 shutdown_connections 2024-09-21T01:28:44.839 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.824+0000 7f87ace7f640 1 -- 172.21.15.45:0/1203824522 wait complete. 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87ace7f640 1 Processor -- start 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87ace7f640 1 -- start start 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87ace7f640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a81023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87ace7f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a8104ff0 0x7f87a8102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87ace7f640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81108f0 con 0x7f87a81046a0 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87ace7f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f87a8110770 con 0x7f87a8104ff0 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.825+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a81023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a81023e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42220/0 (socket says 172.21.15.45:42220) 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87a659c640 1 -- 172.21.15.45:0/2793621153 learned_addr learned my addr 172.21.15.45:0/2793621153 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f879dd9b640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a8104ff0 0x7f87a8102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87a659c640 1 -- 172.21.15.45:0/2793621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a8104ff0 msgr2=0x7f87a8102920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87a659c640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a8104ff0 0x7f87a8102920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87a659c640 1 -- 172.21.15.45:0/2793621153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87a81042c0 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87a659c640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a81023e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f879802a9c0 tx=0x7f8798004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f879ffff640 1 -- 172.21.15.45:0/2793621153 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f879803f070 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87a8102f20 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f879ffff640 1 -- 172.21.15.45:0/2793621153 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f87980051c0 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f879ffff640 1 -- 172.21.15.45:0/2793621153 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798035030 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.826+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87a81a5bd0 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.829+0000 7f879ffff640 1 -- 172.21.15.45:0/2793621153 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f8798042070 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.829+0000 7f879ffff640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f879407af60 0x7f879407d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.829+0000 7f87b33f3740 1 Processor -- start 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f879ffff640 1 -- 172.21.15.45:0/2793621153 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7f87980bee50 con 0x7f87a81046a0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f879dd9b640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f879407af60 0x7f879407d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87b33f3740 1 -- start start 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x560695709e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069570a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.841 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87b33f3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560695681d00 con 0x56069533b210 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87b33f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560695671130 con 0x5606953640d0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f879dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069570a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f879dd9b640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069570a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42226/0 (socket says 172.21.15.45:42226) 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f879dd9b640 1 -- 172.21.15.45:0/951584663 learned_addr learned my addr 172.21.15.45:0/951584663 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87a659c640 1 --2- 172.21.15.45:0/951584663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x560695709e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f879dd9b640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f879407af60 0x7f879407d470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f87a81067d0 tx=0x7f879000b410 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87a659c640 1 -- 172.21.15.45:0/951584663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 msgr2=0x56069570a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87a659c640 1 --2- 172.21.15.45:0/951584663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069570a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.830+0000 7f87a659c640 1 -- 172.21.15.45:0/951584663 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606953cf6b0 con 0x5606953640d0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.831+0000 7f87a659c640 1 --2- 172.21.15.45:0/951584663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x560695709e80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8798085260 tx=0x7f879802a4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2146d79a5ecec5f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.831+0000 7f879cd99640 1 -- 172.21.15.45:0/951584663 <== mon.1 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798049070 con 0x5606953640d0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.831+0000 7f879cd99640 1 -- 172.21.15.45:0/951584663 <== mon.1 v2:172.21.15.90:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f879808ee40 con 0x5606953640d0 2024-09-21T01:28:44.842 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.831+0000 7f879cd99640 1 -- 172.21.15.45:0/951584663 <== mon.1 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798003c90 con 0x5606953640d0 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 -- 172.21.15.45:0/951584663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 msgr2=0x560695709e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 --2- 172.21.15.45:0/951584663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x560695709e80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8798085260 tx=0x7f879802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 -- 172.21.15.45:0/951584663 shutdown_connections 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 --2- 172.21.15.45:0/951584663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069570a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 --2- 172.21.15.45:0/951584663 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x560695709e80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 -- 172.21.15.45:0/951584663 >> 172.21.15.45:0/951584663 conn(0x5606953584c0 msgr2=0x560695707e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 -- 172.21.15.45:0/951584663 shutdown_connections 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 -- 172.21.15.45:0/951584663 wait complete. 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 -- start start 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.832+0000 7f87b33f3740 1 Processor -- start 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695715bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x5606957161b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87b33f3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x56069560c100 con 0x56069533b210 2024-09-21T01:28:44.843 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87b33f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x560695674f40 con 0x5606953640d0 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87b33f3740 10 client.0 _ll_register_callbacks cb 0x56069566b430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695715bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695715bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42238/0 (socket says 172.21.15.45:42238) 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 -- 172.21.15.45:0/271359884 learned_addr learned my addr 172.21.15.45:0/271359884 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f879dd9b640 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x5606957161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 -- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 msgr2=0x5606957161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x5606957161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 -- 172.21.15.45:0/271359884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560695639af0 con 0x56069533b210 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87a659c640 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695715bd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f87980854f0 tx=0x7f8798081a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bddb674027477c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f878dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f878dffb640 1 -- 172.21.15.45:0/271359884 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798049050 con 0x56069533b210 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f878dffb640 1 -- 172.21.15.45:0/271359884 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7f8798003b10 con 0x56069533b210 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.833+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f879808ee40 con 0x56069533b210 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f878dffb640 1 -- 172.21.15.45:0/271359884 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f87980847c0 con 0x56069533b210 2024-09-21T01:28:44.844 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 10 client.14674 fetch_fsmap learned FSMap version 14 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) -- 0x560695721990 con 0x56069533b210 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f878dffb640 1 -- 172.21.15.45:0/271359884 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f879808ee40 con 0x56069533b210 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f878dffb640 1 -- 172.21.15.45:0/271359884 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 14) ==== 4216+0+0 (secure 0 0 0) 0x7f8798003710 con 0x56069533b210 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 10 client.14674 fetch_fsmap finished waiting for FSMap version 14 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 10 client.14674 resolve_mds: resolved name 'foo.smithi090.savegz' to daemon mds.foo.smithi090.savegz 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 20 client.14674 populate_metadata read hostname 'smithi045' 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953433f0 0x560695722610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 4 client.14674 mds_command: new command op to 24393 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 --> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560695683c50 con 0x5606953433f0 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.834+0000 7f879dd9b640 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953433f0 0x560695722610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.835+0000 7f879dd9b640 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953433f0 0x560695722610 crc :-1 s=READY pgs=4 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=fedd0f054a99ffa7 server_cookie=bdfd8be629e43434 in_seq=0 out_seq=0 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.835+0000 7f878dffb640 10 client.14674 ms_handle_connect on v2:172.21.15.90:6832/1256658252 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f878dffb640 1 -- 172.21.15.45:0/271359884 <== mds.? v2:172.21.15.90:6832/1256658252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24313 (crc 0 0 0) 0x560695683c50 con 0x5606953433f0 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f878dffb640 10 client.14674 handle_command_reply: tid=0 2024-09-21T01:28:44.845 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 client.14674 shutdown 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 msgr2=0x560695715bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695715bd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f87980854f0 tx=0x7f8798081a00 comp rx=0 tx=0).stop 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 shutdown_connections 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953433f0 0x560695722610 crc :-1 s=READY pgs=4 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953640d0 0x5606957161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f878dffb640 0 client.14674 ms_handle_reset on v2:172.21.15.90:6832/1256658252 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 --2- 172.21.15.45:0/271359884 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695715bd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 >> 172.21.15.45:0/271359884 conn(0x5606953584c0 msgr2=0x56069570aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 shutdown_connections 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 1 -- 172.21.15.45:0/271359884 wait complete. 2024-09-21T01:28:44.846 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.836+0000 7f87b33f3740 20 client.14674 trim_cache size 0 max 16384 2024-09-21T01:28:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.854+0000 7f87b33f3740 1 Processor -- start 2024-09-21T01:28:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.854+0000 7f87b33f3740 1 -- start start 2024-09-21T01:28:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695711110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.856 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069570d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87b33f3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81ae520 con 0x56069533b210 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87b33f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81abe70 con 0x5606953433f0 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695711110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695711110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42242/0 (socket says 172.21.15.45:42242) 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 -- 172.21.15.45:0/936679379 learned_addr learned my addr 172.21.15.45:0/936679379 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f879dd9b640 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069570d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 -- 172.21.15.45:0/936679379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 msgr2=0x56069570d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069570d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 -- 172.21.15.45:0/936679379 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560695639af0 con 0x56069533b210 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f879dd9b640 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069570d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f87a659c640 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695711110 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8798085260 tx=0x7f879802a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6abdf0200e5f71 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.855+0000 7f878ffff640 1 -- 172.21.15.45:0/936679379 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798049070 con 0x56069533b210 2024-09-21T01:28:44.857 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f878ffff640 1 -- 172.21.15.45:0/936679379 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f879808ee40 con 0x56069533b210 2024-09-21T01:28:44.858 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f878ffff640 1 -- 172.21.15.45:0/936679379 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798003ba0 con 0x56069533b210 2024-09-21T01:28:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 -- 172.21.15.45:0/936679379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 msgr2=0x560695711110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695711110 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8798085260 tx=0x7f879802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 -- 172.21.15.45:0/936679379 shutdown_connections 2024-09-21T01:28:44.860 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069570d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 --2- 172.21.15.45:0/936679379 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x560695711110 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 -- 172.21.15.45:0/936679379 >> 172.21.15.45:0/936679379 conn(0x5606953584c0 msgr2=0x56069570b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 -- 172.21.15.45:0/936679379 shutdown_connections 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.856+0000 7f87b33f3740 1 -- 172.21.15.45:0/936679379 wait complete. 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 1 -- start start 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 1 Processor -- start 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069578a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069578a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x560695722490 con 0x56069533b210 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f87a81ac4e0 con 0x5606953433f0 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87b33f3740 10 client.0 _ll_register_callbacks cb 0x7f8798089070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.857+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069578a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87a659c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069578a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42244/0 (socket says 172.21.15.45:42244) 2024-09-21T01:28:44.861 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87a659c640 1 -- 172.21.15.45:0/921499642 learned_addr learned my addr 172.21.15.45:0/921499642 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f879dd9b640 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069578a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87a659c640 1 -- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 msgr2=0x56069578a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87a659c640 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069578a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87a659c640 1 -- 172.21.15.45:0/921499642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560695726630 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87a659c640 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069578a350 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f87980854f0 tx=0x7f8798004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ed5cf65b32698d server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f878dffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f878dffb640 1 -- 172.21.15.45:0/921499642 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f8798049070 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f878dffb640 1 -- 172.21.15.45:0/921499642 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7f8798092030 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f879808ee40 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f878dffb640 1 -- 172.21.15.45:0/921499642 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f879809b080 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87b33f3740 10 client.14678 fetch_fsmap learned FSMap version 14 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=3+}) -- 0x56069578f130 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.858+0000 7f878dffb640 1 -- 172.21.15.45:0/921499642 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f879808ee40 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f878dffb640 1 -- 172.21.15.45:0/921499642 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 14) ==== 4216+0+0 (secure 0 0 0) 0x7f87980a7020 con 0x56069533b210 2024-09-21T01:28:44.862 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f87b33f3740 10 client.14678 fetch_fsmap finished waiting for FSMap version 14 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f87b33f3740 10 client.14678 resolve_mds: resolved name 'foo.smithi090.savegz' to daemon mds.foo.smithi090.savegz 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f87b33f3740 20 client.14678 populate_metadata read hostname 'smithi045' 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f87b33f3740 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953640d0 0x5606957929d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f87b33f3740 4 client.14678 mds_command: new command op to 24393 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 --> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] -- command(tid 0: {"prefix": "status"}) -- 0x56069567f700 con 0x5606953640d0 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f879dd9b640 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953640d0 0x5606957929d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f879dd9b640 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953640d0 0x5606957929d0 crc :-1 s=READY pgs=5 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.? client_cookie=21931fdcfdd5f29f server_cookie=2a1d71c33dba59dd in_seq=0 out_seq=0 2024-09-21T01:28:44.863 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.859+0000 7f878dffb640 10 client.14678 ms_handle_connect on v2:172.21.15.90:6832/1256658252 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f878dffb640 1 -- 172.21.15.45:0/921499642 <== mds.? v2:172.21.15.90:6832/1256658252 1 ==== command_reply(tid 0: 0 ) ==== 8+0+269 (crc 0 0 0) 0x56069567f700 con 0x5606953640d0 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f878dffb640 10 client.14678 handle_command_reply: tid=0 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 client.14678 shutdown 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 msgr2=0x56069578a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069578a350 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f87980854f0 tx=0x7f8798004cd0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 shutdown_connections 2024-09-21T01:28:44.864 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:6832/1256658252,v1:172.21.15.90:6833/1256658252] conn(0x5606953640d0 0x5606957929d0 crc :-1 s=READY pgs=5 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5606953433f0 0x56069578a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 --2- 172.21.15.45:0/921499642 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x56069533b210 0x56069578a350 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 >> 172.21.15.45:0/921499642 conn(0x5606953584c0 msgr2=0x56069570b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f878dffb640 0 client.14678 ms_handle_reset on v2:172.21.15.90:6832/1256658252 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 shutdown_connections 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 1 -- 172.21.15.45:0/921499642 wait complete. 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.860+0000 7f87b33f3740 20 client.14678 trim_cache size 0 max 16384 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9b2fd594-77b7-11ef-baf5-efdc52797490", 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": -1, 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 24393, 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:standby", 2024-09-21T01:28:44.865 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:standby", 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 12, 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 0, 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 0, 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 4.3853116830000003 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f879407af60 msgr2=0x7f879407d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f879407af60 0x7f879407d470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f87a81067d0 tx=0x7f879000b410 comp rx=0 tx=0).stop 2024-09-21T01:28:44.866 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 msgr2=0x7f87a81023e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a81023e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f879802a9c0 tx=0x7f8798004c60 comp rx=0 tx=0).stop 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 shutdown_connections 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f879407af60 0x7f879407d470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f87a8104ff0 0x7f87a8102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 --2- 172.21.15.45:0/2793621153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87a81046a0 0x7f87a81023e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 >> 172.21.15.45:0/2793621153 conn(0x7f87a80fe1a0 msgr2=0x7f87a810cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 shutdown_connections 2024-09-21T01:28:44.867 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:44.861+0000 7f87ace7f640 1 -- 172.21.15.45:0/2793621153 wait complete. 2024-09-21T01:28:45.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:45 smithi045 ceph-mon[28361]: pgmap v227: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1 op/s 2024-09-21T01:28:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:45.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:45 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:45 smithi090 ceph-mon[35142]: pgmap v227: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1 op/s 2024-09-21T01:28:45.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:45.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:45 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:47.239 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph fs dump --format=json 2024-09-21T01:28:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:47 smithi045 ceph-mon[28361]: pgmap v228: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:28:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:47 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:active 2024-09-21T01:28:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:47 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=bar.smithi045.tspicn=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2 up:standby 2024-09-21T01:28:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:47 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:47.985 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:47 smithi090 ceph-mon[35142]: pgmap v228: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-21T01:28:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:47 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] up:active 2024-09-21T01:28:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:47 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=bar.smithi045.tspicn=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2 up:standby 2024-09-21T01:28:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:47 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 -- 172.21.15.45:0/3507577049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104fc0 msgr2=0x7f944010d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 --2- 172.21.15.45:0/3507577049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104fc0 0x7f944010d600 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9428009880 tx=0x7f942802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 -- 172.21.15.45:0/3507577049 shutdown_connections 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 --2- 172.21.15.45:0/3507577049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104fc0 0x7f944010d600 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 --2- 172.21.15.45:0/3507577049 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9440104590 0x7f94401049a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 -- 172.21.15.45:0/3507577049 >> 172.21.15.45:0/3507577049 conn(0x7f94400fe1a0 msgr2=0x7f94401005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.767+0000 7f9444c55640 1 -- 172.21.15.45:0/3507577049 shutdown_connections 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.768+0000 7f9444c55640 1 -- 172.21.15.45:0/3507577049 wait complete. 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.768+0000 7f9444c55640 1 Processor -- start 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.768+0000 7f9444c55640 1 -- start start 2024-09-21T01:28:48.777 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f9444c55640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 0x7f9440199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f9444c55640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9440104fc0 0x7f9440199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f9444c55640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f9440110920 con 0x7f9440104590 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f9444c55640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f94401107a0 con 0x7f9440104fc0 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 0x7f9440199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 0x7f9440199950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37180/0 (socket says 172.21.15.45:37180) 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 -- 172.21.15.45:0/2027893712 learned_addr learned my addr 172.21.15.45:0/2027893712 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f9435d9b640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9440104fc0 0x7f9440199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 -- 172.21.15.45:0/2027893712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9440104fc0 msgr2=0x7f9440199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9440104fc0 0x7f9440199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 -- 172.21.15.45:0/2027893712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f944019f190 con 0x7f9440104590 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.769+0000 7f943e59c640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 0x7f9440199950 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f943000b420 tx=0x7f943000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.770+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9430011630 con 0x7f9440104590 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.770+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f9430016070 con 0x7f9440104590 2024-09-21T01:28:48.778 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.770+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f944019f480 con 0x7f9440104590 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.770+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f9430010b80 con 0x7f9440104590 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.770+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f944019fa60 con 0x7f9440104590 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.771+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f944006bd30 con 0x7f9440104590 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.772+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f94300117d0 con 0x7f9440104590 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.772+0000 7f9437fff640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f942c07af30 0x7f942c07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.772+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7f943009dd30 con 0x7f9440104590 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.772+0000 7f9435d9b640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f942c07af30 0x7f942c07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.773+0000 7f9435d9b640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f942c07af30 0x7f942c07d440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9428000c00 tx=0x7f9428005780 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:48.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:48.777+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f9430068e70 con 0x7f9440104590 2024-09-21T01:28:48.846 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:48 smithi045 ceph-mon[28361]: mds.? [v2:172.21.15.90:6834/764561033,v1:172.21.15.90:6835/764561033] up:standby 2024-09-21T01:28:48.846 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:48 smithi045 ceph-mon[28361]: fsmap foo:1 bar:1 {bar:0=bar.smithi045.tspicn=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2 up:standby 2024-09-21T01:28:48.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:48 smithi090 ceph-mon[35142]: mds.? [v2:172.21.15.90:6834/764561033,v1:172.21.15.90:6835/764561033] up:standby 2024-09-21T01:28:48.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:48 smithi090 ceph-mon[35142]: fsmap foo:1 bar:1 {bar:0=bar.smithi045.tspicn=up:active,foo:0=foo.smithi045.qpbfpk=up:active} 2 up:standby 2024-09-21T01:28:49.019 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.018+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9440104a20 con 0x7f9440104590 2024-09-21T01:28:49.020 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.019+0000 7f9437fff640 1 -- 172.21.15.45:0/2027893712 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6611 (secure 0 0 0) 0x7f943006dd20 con 0x7f9440104590 2024-09-21T01:28:49.020 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:28:49.020 INFO:teuthology.orchestra.run.smithi045.stdout:{"epoch":16,"btime":"2024-09-21T01:28:47:582182+0000","default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24393,"name":"foo.smithi090.savegz","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.90:6833/1256658252","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1256658252},{"type":"v1","addr":"172.21.15.90:6833","nonce":1256658252}]},"join_fscid":1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24403,"name":"bar.smithi090.qzpqdi","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.90:6835/764561033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":764561033},{"type":"v1","addr":"172.21.15.90:6835","nonce":764561033}]},"join_fscid":2,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":16}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:22.773074+0000","modified":"2024-09-21T01:28:30.825886+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14620},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14620":{"gid":14620,"name":"foo.smithi045.qpbfpk","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.45:6835/4020685920","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6834","nonce":4020685920},{"type":"v1","addr":"172.21.15.45:6835","nonce":4020685920}]},"join_fscid":1,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14620,"qdb_cluster":[14620]},"id":1},{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-21T01:28:27.820077+0000","modified":"2024-09-21T01:28:46.439903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":14652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14652":{"gid":14652,"name":"bar.smithi045.tspicn","rank":0,"incarnation":11,"state":"up:active","state_seq":5,"addr":"172.21.15.45:6837/1900773137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:6836","nonce":1900773137},{"type":"v1","addr":"172.21.15.45:6837","nonce":1900773137}]},"join_fscid":2,"export_targets":[],"features":4540719139924082687,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":14652,"qdb_cluster":[14652]},"id":2}]} 2024-09-21T01:28:49.020 INFO:teuthology.orchestra.run.smithi045.stderr:dumped fsmap epoch 16 2024-09-21T01:28:49.022 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f942c07af30 msgr2=0x7f942c07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:49.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f942c07af30 0x7f942c07d440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9428000c00 tx=0x7f9428005780 comp rx=0 tx=0).stop 2024-09-21T01:28:49.023 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 msgr2=0x7f9440199950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:49.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 0x7f9440199950 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f943000b420 tx=0x7f943000b8e0 comp rx=0 tx=0).stop 2024-09-21T01:28:49.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 shutdown_connections 2024-09-21T01:28:49.026 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f942c07af30 0x7f942c07d440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:49.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9440104fc0 0x7f9440199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:49.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 --2- 172.21.15.45:0/2027893712 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9440104590 0x7f9440199950 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:49.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.021+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 >> 172.21.15.45:0/2027893712 conn(0x7f94400fe1a0 msgr2=0x7f9440108d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:49.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.022+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 shutdown_connections 2024-09-21T01:28:49.027 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:49.022+0000 7f9444c55640 1 -- 172.21.15.45:0/2027893712 wait complete. 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: pgmap v229: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2027893712' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:49.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.671 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:49 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-21T01:28:27.820077+0000', 'modified': '2024-09-21T01:28:46.439903+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 63, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14652}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14652': {'gid': 14652, 'name': 'bar.smithi045.tspicn', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.45:6837/1900773137', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.45:6836', 'nonce': 1900773137}, {'type': 'v1', 'addr': '172.21.15.45:6837', 'nonce': 1900773137}]}, 'join_fscid': 2, 'export_targets': [], 'features': 4540719139924082687, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'bar', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 14652, 'qdb_cluster': [14652]} 2024-09-21T01:28:49.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-21T01:28:49.752 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph tell mds.bar.smithi045.tspicn status 2024-09-21T01:28:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: pgmap v229: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2027893712' entity='client.admin' cmd={"prefix": "fs dump", "format": "json"} : dispatch 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:28:49.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:49 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:50.012 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 -- 172.21.15.45:0/3406106134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0102750 msgr2=0x7fe5f0102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3406106134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0102750 0x7fe5f0102b60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009880 tx=0x7fe5e002a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 -- 172.21.15.45:0/3406106134 shutdown_connections 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3406106134 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f0108090 0x7fe5f01084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3406106134 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0102750 0x7fe5f0102b60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 -- 172.21.15.45:0/3406106134 >> 172.21.15.45:0/3406106134 conn(0x7fe5f00fe1a0 msgr2=0x7fe5f01005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:50.464 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 -- 172.21.15.45:0/3406106134 shutdown_connections 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.460+0000 7fe5f4def640 1 -- 172.21.15.45:0/3406106134 wait complete. 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5f4def640 1 Processor -- start 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5f4def640 1 -- start start 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5f4def640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f0102750 0x7fe5f01984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5f4def640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 0x7fe5f0198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5f4def640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f010f820 con 0x7fe5f0108090 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5f4def640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f010f6a0 con 0x7fe5f0102750 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 0x7fe5f0198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.461+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 0x7fe5f0198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37204/0 (socket says 172.21.15.45:37204) 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3269139622 learned_addr learned my addr 172.21.15.45:0/3269139622 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f0102750 0x7fe5f01984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.465 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3269139622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f0102750 msgr2=0x7fe5f01984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f0102750 0x7fe5f01984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e7fff640 1 -- 172.21.15.45:0/3269139622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f006b990 con 0x7fe5f0108090 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 0x7fe5f0198a00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c8f0 tx=0x7fe5d800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e77fe640 1 -- 172.21.15.45:0/3269139622 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d8004020 con 0x7fe5f0108090 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f0198f40 con 0x7fe5f0108090 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e77fe640 1 -- 172.21.15.45:0/3269139622 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5d8007e40 con 0x7fe5f0108090 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5e77fe640 1 -- 172.21.15.45:0/3269139622 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d80047a0 con 0x7fe5f0108090 2024-09-21T01:28:50.466 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.462+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f01a5bb0 con 0x7fe5f0108090 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.464+0000 7fe5e77fe640 1 -- 172.21.15.45:0/3269139622 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fe5d80079b0 con 0x7fe5f0108090 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.465+0000 7fe5e77fe640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe5dc07af30 0x7fe5dc07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.465+0000 7fe5e77fe640 1 -- 172.21.15.45:0/3269139622 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7fe5d809e370 con 0x7fe5f0108090 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.465+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe5dc07af30 0x7fe5dc07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.465+0000 7fe5fb363740 1 Processor -- start 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.465+0000 7fe5fb363740 1 -- start start 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.465+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe5dc07af30 0x7fe5dc07d440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0006010 tx=0x7fe5e00058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:50.468 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b352e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b3533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5fb363740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f26b2cad00 con 0x55f26af84210 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5fb363740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f26b2ba130 con 0x55f26afad0d0 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b3533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b3533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37218/0 (socket says 172.21.15.45:37218) 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 -- 172.21.15.45:0/1810076535 learned_addr learned my addr 172.21.15.45:0/1810076535 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/1810076535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b352e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 -- 172.21.15.45:0/1810076535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 msgr2=0x55f26b352e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/1810076535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b352e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 -- 172.21.15.45:0/1810076535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26b0186b0 con 0x55f26af84210 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/1810076535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b3533c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8060fb0 tx=0x7fe5d8061160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d00a3c9960940c server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.466+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/1810076535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d801d070 con 0x55f26af84210 2024-09-21T01:28:50.469 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/1810076535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5d80644f0 con 0x55f26af84210 2024-09-21T01:28:50.470 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5e4ff9640 1 -- 172.21.15.45:0/1810076535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d8069170 con 0x55f26af84210 2024-09-21T01:28:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5fb363740 1 -- 172.21.15.45:0/1810076535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 msgr2=0x55f26b3533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5fb363740 1 --2- 172.21.15.45:0/1810076535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b3533c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe5d8060fb0 tx=0x7fe5d8061160 comp rx=0 tx=0).stop 2024-09-21T01:28:50.473 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5fb363740 1 -- 172.21.15.45:0/1810076535 shutdown_connections 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5fb363740 1 --2- 172.21.15.45:0/1810076535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b3533c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5fb363740 1 --2- 172.21.15.45:0/1810076535 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b352e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.467+0000 7fe5fb363740 1 -- 172.21.15.45:0/1810076535 >> 172.21.15.45:0/1810076535 conn(0x55f26afa14c0 msgr2=0x55f26b350e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 -- 172.21.15.45:0/1810076535 shutdown_connections 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 -- 172.21.15.45:0/1810076535 wait complete. 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 -- start start 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 Processor -- start 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b35ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b35f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f26b255100 con 0x55f26afad0d0 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x55f26b2bdf40 con 0x55f26af84210 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5fb363740 10 client.0 _ll_register_callbacks cb 0x55f26b2b4430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:50.474 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.468+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b35f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b35f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37222/0 (socket says 172.21.15.45:37222) 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5e7fff640 1 -- 172.21.15.45:0/4112718300 learned_addr learned my addr 172.21.15.45:0/4112718300 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b35ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5e7fff640 1 -- 172.21.15.45:0/4112718300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 msgr2=0x55f26b35ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b35ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5e7fff640 1 -- 172.21.15.45:0/4112718300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26b282af0 con 0x55f26afad0d0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b35ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b35f220 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80057a0 tx=0x7fe5d800aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c77874164e7e32 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/4112718300 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d801d040 con 0x55f26afad0d0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/4112718300 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7fe5d809e110 con 0x55f26afad0d0 2024-09-21T01:28:50.475 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d80644f0 con 0x55f26afad0d0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/4112718300 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fe5d8084050 con 0x55f26afad0d0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5fb363740 10 client.14690 fetch_fsmap learned FSMap version 16 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.469+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=3+}) -- 0x55f26b36aa00 con 0x55f26afad0d0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/4112718300 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5d80644f0 con 0x55f26afad0d0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/4112718300 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 16) ==== 4216+0+0 (secure 0 0 0) 0x7fe5d807aa50 con 0x55f26afad0d0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5fb363740 10 client.14690 fetch_fsmap finished waiting for FSMap version 16 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5fb363740 10 client.14690 resolve_mds: resolved name 'bar.smithi045.tspicn' to daemon mds.bar.smithi045.tspicn 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5fb363740 20 client.14690 populate_metadata read hostname 'smithi045' 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5fb363740 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af8c3f0 0x55f26b36b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5fb363740 4 client.14690 mds_command: new command op to 14652 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 --> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f26b2ccc50 con 0x55f26af8c3f0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af8c3f0 0x55f26b36b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5d5ffb640 10 client.14690 ms_handle_connect on v2:172.21.15.45:6836/1900773137 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.470+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af8c3f0 0x55f26b36b840 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40f2d04606c54f74 server_cookie=b8b5e9fffb566ea8 in_seq=0 out_seq=0 2024-09-21T01:28:50.476 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/4112718300 <== mds.0 v2:172.21.15.45:6836/1900773137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24444 (crc 0 0 0) 0x55f26b2ccc50 con 0x55f26af8c3f0 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5d5ffb640 10 client.14690 handle_command_reply: tid=0 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 client.14690 shutdown 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 msgr2=0x55f26b35f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b35f220 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80057a0 tx=0x7fe5d800aea0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 shutdown_connections 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af8c3f0 0x55f26b36b840 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b35f220 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 --2- 172.21.15.45:0/4112718300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b35ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 >> 172.21.15.45:0/4112718300 conn(0x55f26afa14c0 msgr2=0x55f26b353eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5d5ffb640 0 client.14690 ms_handle_reset on v2:172.21.15.45:6836/1900773137 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 shutdown_connections 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.471+0000 7fe5fb363740 1 -- 172.21.15.45:0/4112718300 wait complete. 2024-09-21T01:28:50.477 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.472+0000 7fe5fb363740 20 client.14690 trim_cache size 0 max 16384 2024-09-21T01:28:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5fb363740 1 Processor -- start 2024-09-21T01:28:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5fb363740 1 -- start start 2024-09-21T01:28:50.491 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b352dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b359030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5fb363740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01ae6d0 con 0x55f26afad0d0 2024-09-21T01:28:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5fb363740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01ac020 con 0x55f26af84210 2024-09-21T01:28:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b359030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5e7fff640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b359030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37226/0 (socket says 172.21.15.45:37226) 2024-09-21T01:28:50.492 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.490+0000 7fe5e7fff640 1 -- 172.21.15.45:0/182607816 learned_addr learned my addr 172.21.15.45:0/182607816 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5e7fff640 1 -- 172.21.15.45:0/182607816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 msgr2=0x55f26b352dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/182607816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b352dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5e7fff640 1 -- 172.21.15.45:0/182607816 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f26b282af0 con 0x55f26afad0d0 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/182607816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b359030 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80646e0 tx=0x7fe5d8088070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41a375e01e92fcad server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5d7fff640 1 -- 172.21.15.45:0/182607816 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d801d040 con 0x55f26afad0d0 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5d7fff640 1 -- 172.21.15.45:0/182607816 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5d804f020 con 0x55f26afad0d0 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.491+0000 7fe5d7fff640 1 -- 172.21.15.45:0/182607816 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5d807aa50 con 0x55f26afad0d0 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 -- 172.21.15.45:0/182607816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 msgr2=0x55f26b359030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.493 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 --2- 172.21.15.45:0/182607816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b359030 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5d80646e0 tx=0x7fe5d8088070 comp rx=0 tx=0).stop 2024-09-21T01:28:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 -- 172.21.15.45:0/182607816 shutdown_connections 2024-09-21T01:28:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 --2- 172.21.15.45:0/182607816 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26afad0d0 0x55f26b359030 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 --2- 172.21.15.45:0/182607816 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26af84210 0x55f26b352dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 -- 172.21.15.45:0/182607816 >> 172.21.15.45:0/182607816 conn(0x55f26afa14c0 msgr2=0x55f26b35d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 -- 172.21.15.45:0/182607816 shutdown_connections 2024-09-21T01:28:50.494 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 -- 172.21.15.45:0/182607816 wait complete. 2024-09-21T01:28:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 -- start start 2024-09-21T01:28:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.492+0000 7fe5fb363740 1 Processor -- start 2024-09-21T01:28:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b36d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5fb363740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b36dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5fb363740 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x55f26b36b6c0 con 0x55f26af84210 2024-09-21T01:28:50.497 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5fb363740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f01ac690 con 0x55f26afad0d0 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5fb363740 10 client.0 _ll_register_callbacks cb 0x7fe5d8068070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b36d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b36d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37234/0 (socket says 172.21.15.45:37234) 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 -- 172.21.15.45:0/3724199706 learned_addr learned my addr 172.21.15.45:0/3724199706 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b36dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 -- 172.21.15.45:0/3724199706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 msgr2=0x55f26b36dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b36dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 -- 172.21.15.45:0/3724199706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f26b355b90 con 0x55f26af84210 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5ee59c640 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b36d6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe5e002a9c0 tx=0x7fe5e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f8a02db5d94031 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.45:3300/0 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/3724199706 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fe5e0035050 con 0x55f26af84210 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5d804f020 con 0x55f26af84210 2024-09-21T01:28:50.498 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.493+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/3724199706 <== mon.0 v2:172.21.15.45:3300/0 2 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7fe5e0058040 con 0x55f26af84210 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/3724199706 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fe5e0064050 con 0x55f26af84210 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 10 client.14694 fetch_fsmap learned FSMap version 16 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=3+}) -- 0x55f26b373950 con 0x55f26af84210 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/3724199706 <== mon.0 v2:172.21.15.45:3300/0 4 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fe5d804f020 con 0x55f26af84210 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/3724199706 <== mon.0 v2:172.21.15.45:3300/0 5 ==== fsmap(e 16) ==== 4216+0+0 (secure 0 0 0) 0x7fe5e0067020 con 0x55f26af84210 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 10 client.14694 fetch_fsmap finished waiting for FSMap version 16 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 10 client.14694 resolve_mds: resolved name 'bar.smithi045.tspicn' to daemon mds.bar.smithi045.tspicn 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 20 client.14694 populate_metadata read hostname 'smithi045' 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af6ba50 0x55f26b3db6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 4 client.14694 mds_command: new command op to 14652 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-21T01:28:50.499 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 --> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] -- command(tid 0: {"prefix": "status"}) -- 0x55f26b2c8700 con 0x55f26af6ba50 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af6ba50 0x55f26b3db6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5d5ffb640 10 client.14694 ms_handle_connect on v2:172.21.15.45:6836/1900773137 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.494+0000 7fe5e7fff640 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af6ba50 0x55f26b3db6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ce98d4ffed37a7e server_cookie=5fdb8a670d4a6158 in_seq=0 out_seq=0 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5d5ffb640 1 -- 172.21.15.45:0/3724199706 <== mds.0 v2:172.21.15.45:6836/1900773137 1 ==== command_reply(tid 0: 0 ) ==== 8+0+329 (crc 0 0 0) 0x55f26b2c8700 con 0x55f26af6ba50 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5d5ffb640 10 client.14694 handle_command_reply: tid=0 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 client.14694 shutdown 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 msgr2=0x55f26b36d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b36d6f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe5e002a9c0 tx=0x7fe5e0009260 comp rx=0 tx=0).stop 2024-09-21T01:28:50.500 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 shutdown_connections 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:6836/1900773137,v1:172.21.15.45:6837/1900773137] conn(0x55f26af6ba50 0x55f26b3db6a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x55f26afad0d0 0x55f26b36dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 --2- 172.21.15.45:0/3724199706 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x55f26af84210 0x55f26b36d6f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 >> 172.21.15.45:0/3724199706 conn(0x55f26afa14c0 msgr2=0x55f26b362000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5d5ffb640 0 client.14694 ms_handle_reset on v2:172.21.15.45:6836/1900773137 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 shutdown_connections 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 1 -- 172.21.15.45:0/3724199706 wait complete. 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.495+0000 7fe5fb363740 20 client.14694 trim_cache size 0 max 16384 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "cluster_fsid": "9b2fd594-77b7-11ef-baf5-efdc52797490", 2024-09-21T01:28:50.501 INFO:teuthology.orchestra.run.smithi045.stdout: "whoami": 0, 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 14652, 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "want_state": "up:active", 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "state": "up:active", 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "bar", 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "rank_uptime": 10.079437452000001, 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "mdsmap_epoch": 15, 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch": 63, 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "osdmap_epoch_barrier": 63, 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout: "uptime": 10.196319963000001 2024-09-21T01:28:50.502 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe5dc07af30 msgr2=0x7fe5dc07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe5dc07af30 0x7fe5dc07d440 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0006010 tx=0x7fe5e00058d0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 msgr2=0x7fe5f0198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 0x7fe5f0198a00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5d800c8f0 tx=0x7fe5d800cdb0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 shutdown_connections 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fe5dc07af30 0x7fe5dc07d440 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5f0108090 0x7fe5f0198a00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 --2- 172.21.15.45:0/3269139622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe5f0102750 0x7fe5f01984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 >> 172.21.15.45:0/3269139622 conn(0x7fe5f00fe1a0 msgr2=0x7fe5f01005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 shutdown_connections 2024-09-21T01:28:50.503 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:50.496+0000 7fe5f4def640 1 -- 172.21.15.45:0/3269139622 wait complete. 2024-09-21T01:28:51.260 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-21T01:28:51.290 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-21T01:28:51.290 INFO:tasks.ceph_fuse:config is {'client.0': {'cephfs_name': 'foo'}} 2024-09-21T01:28:51.290 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'foo'} 2024-09-21T01:28:51.291 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T01:28:51.291 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:51.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:51.291 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-21T01:28:51.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:51.308 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-09-21T01:28:51.378 INFO:teuthology.orchestra.run.smithi045.stderr:Cannot find device "ceph-brx" 2024-09-21T01:28:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:28:51.379 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-21T01:28:51.379 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-21T01:28:51.380 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T01:28:51.690 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:51.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:51 smithi045 ceph-mon[28361]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:28:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:51 smithi090 ceph-mon[35142]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T01:28:52.152 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 -- 172.21.15.45:0/3981495019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878103850 msgr2=0x7fb878103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/3981495019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878103850 0x7fb878103c60 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb874009880 tx=0x7fb87402a4d0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 -- 172.21.15.45:0/3981495019 shutdown_connections 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/3981495019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878109190 0x7fb8781095a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/3981495019 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878103850 0x7fb878103c60 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 -- 172.21.15.45:0/3981495019 >> 172.21.15.45:0/3981495019 conn(0x7fb8780ff2a0 msgr2=0x7fb8781016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 -- 172.21.15.45:0/3981495019 shutdown_connections 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 -- 172.21.15.45:0/3981495019 wait complete. 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 Processor -- start 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.145+0000 7fb87f3e1640 1 -- start start 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87f3e1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878103850 0x7fb878199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87f3e1640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 0x7fb878199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87f3e1640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fb878110960 con 0x7fb878109190 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87f3e1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fb8781107e0 con 0x7fb878103850 2024-09-21T01:28:52.153 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 0x7fb878199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 0x7fb878199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:37250/0 (socket says 172.21.15.45:37250) 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 -- 172.21.15.45:0/2920959339 learned_addr learned my addr 172.21.15.45:0/2920959339 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87d17d640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878103850 0x7fb878199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 -- 172.21.15.45:0/2920959339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878103850 msgr2=0x7fb878199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878103850 0x7fb878199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 -- 172.21.15.45:0/2920959339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87819f1a0 con 0x7fb878109190 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87d17d640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878103850 0x7fb878199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb87c97c640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 0x7fb878199f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb868006fd0 tx=0x7fb86800b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.146+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb868015020 con 0x7fb878109190 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.147+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fb868011ac0 con 0x7fb878109190 2024-09-21T01:28:52.154 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.147+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87819f490 con 0x7fb878109190 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.147+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fb868010b80 con 0x7fb878109190 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.147+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87819faa0 con 0x7fb878109190 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.148+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878104ad0 con 0x7fb878109190 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.149+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fb868018020 con 0x7fb878109190 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.149+0000 7fb86e7fc640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb86407af80 0x7fb86407d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.149+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 7224+0+0 (secure 0 0 0) 0x7fb86809dd80 con 0x7fb878109190 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.149+0000 7fb87d17d640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb86407af80 0x7fb86407d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:52.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.150+0000 7fb87d17d640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb86407af80 0x7fb86407d490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb874000c00 tx=0x7fb874004200 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:52.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.154+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fb86801f020 con 0x7fb878109190 2024-09-21T01:28:52.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.381+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb87819a5e0 con 0x7fb878109190 2024-09-21T01:28:52.382 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.381+0000 7fb86e7fc640 1 -- 172.21.15.45:0/2920959339 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+148 (secure 0 0 0) 0x7fb868068f40 con 0x7fb878109190 2024-09-21T01:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout:name: foo, metadata pool: cephfs.foo.meta, data pools: [cephfs.foo.data ] 2024-09-21T01:28:52.383 INFO:teuthology.orchestra.run.smithi045.stdout:name: bar, metadata pool: cephfs.bar.meta, data pools: [cephfs.bar.data ] 2024-09-21T01:28:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.383+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb86407af80 msgr2=0x7fb86407d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.383+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb86407af80 0x7fb86407d490 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb874000c00 tx=0x7fb874004200 comp rx=0 tx=0).stop 2024-09-21T01:28:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.383+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 msgr2=0x7fb878199f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:52.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.383+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 0x7fb878199f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb868006fd0 tx=0x7fb86800b860 comp rx=0 tx=0).stop 2024-09-21T01:28:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 shutdown_connections 2024-09-21T01:28:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fb86407af80 0x7fb86407d490 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb878109190 0x7fb878199f00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 --2- 172.21.15.45:0/2920959339 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb878103850 0x7fb878199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:52.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 >> 172.21.15.45:0/2920959339 conn(0x7fb8780ff2a0 msgr2=0x7fb8781016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:52.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 shutdown_connections 2024-09-21T01:28:52.386 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:52.384+0000 7fb87f3e1640 1 -- 172.21.15.45:0/2920959339 wait complete. 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]} : dispatch 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/2920959339' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:52.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]} : dispatch 2024-09-21T01:28:52.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]} : dispatch 2024-09-21T01:28:52.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]} : dispatch 2024-09-21T01:28:52.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]} : dispatch 2024-09-21T01:28:52.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/2920959339' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:28:52.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:28:52.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:28:53.129 INFO:tasks.cephfs.mount:Mounting Ceph FS foo; just confirmed its presence on cluster 2024-09-21T01:28:53.129 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi045.front.sepia.ceph.com 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:self.cephfs_name = foo 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T01:28:53.130 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-21T01:28:53.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:53.131 DEBUG:teuthology.orchestra.run.smithi045:> ip addr 2024-09-21T01:28:53.148 INFO:teuthology.orchestra.run.smithi045.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T01:28:53.148 INFO:teuthology.orchestra.run.smithi045.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 ::1/128 scope host 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a6 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f0 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:6b:fd:a7 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp7s0f1 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:be brd ff:ff:ff:ff:ff:ff 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f0 2024-09-21T01:28:53.149 INFO:teuthology.orchestra.run.smithi045.stdout: inet 172.21.15.45/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-21T01:28:53.150 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft 42250sec preferred_lft 42250sec 2024-09-21T01:28:53.150 INFO:teuthology.orchestra.run.smithi045.stdout: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-09-21T01:28:53.150 INFO:teuthology.orchestra.run.smithi045.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:28:53.150 INFO:teuthology.orchestra.run.smithi045.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:28:53.150 INFO:teuthology.orchestra.run.smithi045.stdout: link/ether 0c:c4:7a:bd:12:bf brd ff:ff:ff:ff:ff:ff 2024-09-21T01:28:53.150 INFO:teuthology.orchestra.run.smithi045.stdout: altname enp1s0f1 2024-09-21T01:28:53.150 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T01:28:53.150 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:28:53.151 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-21T01:28:53.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add name ceph-brx type bridge 2024-09-21T01:28:53.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr flush dev ceph-brx 2024-09-21T01:28:53.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set ceph-brx up 2024-09-21T01:28:53.151 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T01:28:53.151 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-21T01:28:53.239 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:28:53.362 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:28:53.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:53.370 DEBUG:teuthology.orchestra.run.smithi045:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T01:28:53.396 INFO:teuthology.orchestra.run.smithi045.stdout:1 2024-09-21T01:28:53.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:53.398 DEBUG:teuthology.orchestra.run.smithi045:> route 2024-09-21T01:28:53.471 INFO:teuthology.orchestra.run.smithi045.stdout:Kernel IP routing table 2024-09-21T01:28:53.471 INFO:teuthology.orchestra.run.smithi045.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T01:28:53.471 INFO:teuthology.orchestra.run.smithi045.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-21T01:28:53.471 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-21T01:28:53.471 INFO:teuthology.orchestra.run.smithi045.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T01:28:53.472 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:28:53.472 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-21T01:28:53.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-21T01:28:53.472 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-21T01:28:53.474 DEBUG:teuthology.orchestra.run.smithi045:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-21T01:28:53.474 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-21T01:28:53.560 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:28:53.779 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:28:53.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:53.786 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-21T01:28:53.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:53.803 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list-id 2024-09-21T01:28:53.859 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:28:53.859 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-21T01:28:53.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:53.859 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-21T01:28:53.859 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-21T01:28:53.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:53 smithi045 ceph-mon[28361]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:28:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-21T01:28:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-21T01:28:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T01:28:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-21T01:28:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:53 smithi045 ceph-mon[28361]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T01:28:53.949 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:28:53.972 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:28:53.978 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-21T01:28:53.978 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:28:53.978 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-21T01:28:53.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-21T01:28:53.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T01:28:53.978 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-21T01:28:53.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-21T01:28:53.979 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-21T01:28:53.979 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-21T01:28:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:53 smithi090 ceph-mon[35142]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 7.1 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T01:28:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-21T01:28:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-21T01:28:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T01:28:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-21T01:28:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:53 smithi090 ceph-mon[35142]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T01:28:54.068 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:28:54.142 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:28:54.148 DEBUG:teuthology.orchestra.run.smithi045:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:28:54.148 DEBUG:teuthology.orchestra.run.smithi045:> set -e 2024-09-21T01:28:54.148 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set brx.0 up 2024-09-21T01:28:54.149 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T01:28:54.149 DEBUG:teuthology.orchestra.run.smithi045:> ') 2024-09-21T01:28:54.198 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:28:54.220 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:28:54.227 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'foo'} 2024-09-21T01:28:54.227 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:28:54.227 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:54.242 INFO:teuthology.orchestra.run.smithi045.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-21T01:28:54.242 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:28:54.243 DEBUG:teuthology.orchestra.run.smithi045:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:54.299 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-09-21T01:28:54.365 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-09-21T01:28:54.422 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-09-21T01:28:54.423 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/ad670c58f480d9e376cc1c6dba4c0607970d77491dc2bb482754055aa7bbd7c5/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/35d511838eb8060114365597eedc2ef8bad80d74dc65ca62d9657d5e2bf5357e/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/b685952e2e2d878ba2ce6c441b8fd124b8454d248daafd031c38c59f3f31dda0/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/99edfbb781b52f376b34046ccd185f933cb17ff2ab617733cce508cee86ac18a/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/5d145088b4ea1a0ecd81969e5445fbeae0e8294bdd2918360150e022e3c279ff/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/79723413cbceb56e2386ef1d7ea969a989b4ccd832c7c6e9ce3119bcb2b238e9/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d0ffbccd542a013b862dad4964c062f1a5191cda9957ad8bfba3f0368458ef9d/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/9c05e27e6636bda2c12edddeda46732180428a3e53b4f572c0f998f19a0061fe/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a1b6cdebe689e71c429418fe5326524897937b5f37d90de992db69d64fa44060/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/5e89a7de0fbad5d96bbf74acf5b444bb3e989b0f6bac88654a894b9213faee69/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/07e5a5b141b806d2d20c0b579de4ac1744b52f19d9d69ade702121e314cfae92/merged 2024-09-21T01:28:54.424 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/975651874b44ae6cb409a8d281f45f482bf50affe77b4a3d2a0a47dea2d33c73/merged 2024-09-21T01:28:54.425 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/65f1d527baeba808c191f87fbbfe6f352d1b3242aa722cbf123a3082635ab3cc/merged 2024-09-21T01:28:54.425 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/7f9468bc312dfe95d70fe38c47c2bf1b307a698f5a390af2e5ef5bbb0aa68985/merged 2024-09-21T01:28:54.425 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-09-21T01:28:54.425 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:54.425 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:54.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:54.425 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-09-21T01:28:54.477 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T01:28:54.478 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=foo) 2024-09-21T01:28:54.520 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-09-21T01:28:54.547 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-09-21T01:28:54.603 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:28:54.604 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/ad670c58f480d9e376cc1c6dba4c0607970d77491dc2bb482754055aa7bbd7c5/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/35d511838eb8060114365597eedc2ef8bad80d74dc65ca62d9657d5e2bf5357e/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/b685952e2e2d878ba2ce6c441b8fd124b8454d248daafd031c38c59f3f31dda0/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/99edfbb781b52f376b34046ccd185f933cb17ff2ab617733cce508cee86ac18a/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/5d145088b4ea1a0ecd81969e5445fbeae0e8294bdd2918360150e022e3c279ff/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/79723413cbceb56e2386ef1d7ea969a989b4ccd832c7c6e9ce3119bcb2b238e9/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d0ffbccd542a013b862dad4964c062f1a5191cda9957ad8bfba3f0368458ef9d/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/9c05e27e6636bda2c12edddeda46732180428a3e53b4f572c0f998f19a0061fe/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a1b6cdebe689e71c429418fe5326524897937b5f37d90de992db69d64fa44060/merged 2024-09-21T01:28:54.605 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/5e89a7de0fbad5d96bbf74acf5b444bb3e989b0f6bac88654a894b9213faee69/merged 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/07e5a5b141b806d2d20c0b579de4ac1744b52f19d9d69ade702121e314cfae92/merged 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/975651874b44ae6cb409a8d281f45f482bf50affe77b4a3d2a0a47dea2d33c73/merged 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/65f1d527baeba808c191f87fbbfe6f352d1b3242aa722cbf123a3082635ab3cc/merged 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/7f9468bc312dfe95d70fe38c47c2bf1b307a698f5a390af2e5ef5bbb0aa68985/merged 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:54.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:54.606 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-09-21T01:28:54.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:2024-09-21T01:28:54.655+0000 7f12cab1c480 -1 init, newargv = 0x7f12b8056e90 newargc=15 2024-09-21T01:28:54.656 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:ceph-fuse[79628]: starting ceph client 2024-09-21T01:28:54.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:ceph-fuse[79628]: starting fuse 2024-09-21T01:28:54.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:54 smithi045 ceph-mon[28361]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T01:28:54.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:54 smithi090 ceph-mon[35142]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T01:28:55.762 DEBUG:teuthology.orchestra.run.smithi045:> sudo modprobe fuse 2024-09-21T01:28:55.796 DEBUG:teuthology.orchestra.run.smithi045:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/proc 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/sys 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/dev 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/security 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/shm 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/pts 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/run 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/cgroup 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/pstore 2024-09-21T01:28:55.856 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/bpf 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/ 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/selinux 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/hugepages 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/dev/mqueue 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/debug 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/tracing 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/fs/fuse/connections 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/sys/kernel/config 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/1000 2024-09-21T01:28:55.857 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/ceph 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/ad670c58f480d9e376cc1c6dba4c0607970d77491dc2bb482754055aa7bbd7c5/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/35d511838eb8060114365597eedc2ef8bad80d74dc65ca62d9657d5e2bf5357e/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/run/user/0 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/b685952e2e2d878ba2ce6c441b8fd124b8454d248daafd031c38c59f3f31dda0/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/99edfbb781b52f376b34046ccd185f933cb17ff2ab617733cce508cee86ac18a/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/5d145088b4ea1a0ecd81969e5445fbeae0e8294bdd2918360150e022e3c279ff/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/79723413cbceb56e2386ef1d7ea969a989b4ccd832c7c6e9ce3119bcb2b238e9/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/d0ffbccd542a013b862dad4964c062f1a5191cda9957ad8bfba3f0368458ef9d/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/9c05e27e6636bda2c12edddeda46732180428a3e53b4f572c0f998f19a0061fe/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/a1b6cdebe689e71c429418fe5326524897937b5f37d90de992db69d64fa44060/merged 2024-09-21T01:28:55.858 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/5e89a7de0fbad5d96bbf74acf5b444bb3e989b0f6bac88654a894b9213faee69/merged 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/07e5a5b141b806d2d20c0b579de4ac1744b52f19d9d69ade702121e314cfae92/merged 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/975651874b44ae6cb409a8d281f45f482bf50affe77b4a3d2a0a47dea2d33c73/merged 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/65f1d527baeba808c191f87fbbfe6f352d1b3242aa722cbf123a3082635ab3cc/merged 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/var/lib/containers/storage/overlay/7f9468bc312dfe95d70fe38c47c2bf1b307a698f5a390af2e5ef5bbb0aa68985/merged 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:28:55.859 INFO:teuthology.orchestra.run.smithi045.stdout:/home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:55.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:55.860 DEBUG:teuthology.orchestra.run.smithi045:> ls /sys/fs/fuse/connections 2024-09-21T01:28:55.918 INFO:teuthology.orchestra.run.smithi045.stdout:168 2024-09-21T01:28:55.919 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [168] 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> sudo stdin-killer -- python3 -c ' 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> import glob 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> import re 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> import os 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> import subprocess 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> def _find_admin_socket(client_name): 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> files = glob.glob(asok_path) 2024-09-21T01:28:55.919 DEBUG:teuthology.orchestra.run.smithi045:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> # Given a non-glob path, it better be there 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> if "*" not in asok_path: 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> assert(len(files) == 1) 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> return files[0] 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> for f in files: 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> contents = proc_f.read() 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> if mountpoint in contents: 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> return f 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> print(_find_admin_socket("client.0")) 2024-09-21T01:28:55.920 DEBUG:teuthology.orchestra.run.smithi045:> ' 2024-09-21T01:28:55.921 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:55 smithi045 ceph-mon[28361]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 5 op/s 2024-09-21T01:28:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:55 smithi090 ceph-mon[35142]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 5 op/s 2024-09-21T01:28:56.008 INFO:teuthology.orchestra.run.smithi045.stdout:/var/run/ceph/ceph-client.0.79628.asok 2024-09-21T01:28:56.011 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:28:56.019 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.79628.asok 2024-09-21T01:28:56.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:56.019 DEBUG:teuthology.orchestra.run.smithi045:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.79628.asok status 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout:{ 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "metadata": { 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "entity_id": "0", 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "hostname": "smithi045", 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "pid": "79628", 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "root": "/" 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_count": 0, 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "dentry_pinned_count": 0, 2024-09-21T01:28:56.167 INFO:teuthology.orchestra.run.smithi045.stdout: "id": 14702, 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "inst": { 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "name": { 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "client", 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "num": 14702 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 828761908 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: } 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": { 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "type": "v1", 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "addr": "192.168.0.1:0", 2024-09-21T01:28:56.168 INFO:teuthology.orchestra.run.smithi045.stdout: "nonce": 828761908 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: }, 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "inst_str": "client.14702 192.168.0.1:0/828761908", 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "addr_str": "192.168.0.1:0/828761908", 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "inode_count": 1, 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "mds_epoch": 9, 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch": 65, 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "osd_epoch_barrier": 0, 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "blocklisted": false, 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout: "fs_name": "foo" 2024-09-21T01:28:56.169 INFO:teuthology.orchestra.run.smithi045.stdout:} 2024-09-21T01:28:56.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:56.176 DEBUG:teuthology.orchestra.run.smithi045:> stat --file-system '--printf=%T 2024-09-21T01:28:56.176 DEBUG:teuthology.orchestra.run.smithi045:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:56.192 INFO:teuthology.orchestra.run.smithi045.stdout:fuseblk 2024-09-21T01:28:56.192 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:56.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:56.192 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:56.258 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-21T01:28:56.267 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-21T01:28:56.267 INFO:tasks.ceph_fuse:config is {'client.1': {'cephfs_name': 'bar'}} 2024-09-21T01:28:56.267 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'bar'} 2024-09-21T01:28:56.267 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-21T01:28:56.268 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-21T01:28:56.268 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:28:56.268 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:28:56.315 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:28:56.315 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:28:56.373 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:28:56.676 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:57.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 -- 172.21.15.45:0/3340861556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 msgr2=0x7f3fc41084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:57.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/3340861556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc41084a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009880 tx=0x7f3fb802a4a0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 -- 172.21.15.45:0/3340861556 shutdown_connections 2024-09-21T01:28:57.155 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/3340861556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc41084a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/3340861556 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fc4102750 0x7f3fc4102b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 -- 172.21.15.45:0/3340861556 >> 172.21.15.45:0/3340861556 conn(0x7f3fc40fe1a0 msgr2=0x7f3fc41005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 -- 172.21.15.45:0/3340861556 shutdown_connections 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 -- 172.21.15.45:0/3340861556 wait complete. 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.150+0000 7f3fca1cd640 1 Processor -- start 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fca1cd640 1 -- start start 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fca1cd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fc4102750 0x7f3fc410d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fca1cd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc410d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fca1cd640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc41108c0 con 0x7f3fc4108090 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fca1cd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc4110740 con 0x7f3fc4102750 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fc2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc410d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fc2ffd640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc410d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33468/0 (socket says 172.21.15.45:33468) 2024-09-21T01:28:57.156 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fc2ffd640 1 -- 172.21.15.45:0/1829453205 learned_addr learned my addr 172.21.15.45:0/1829453205 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fc37fe640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fc4102750 0x7f3fc410d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fc2ffd640 1 -- 172.21.15.45:0/1829453205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fc4102750 msgr2=0x7f3fc410d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.151+0000 7f3fc2ffd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fc4102750 0x7f3fc410d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fc2ffd640 1 -- 172.21.15.45:0/1829453205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc410f200 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fc2ffd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc410d860 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3fb802a9c0 tx=0x7f3fb80045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3fb803f070 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f3fb80049d0 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc410de60 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f3fb8035030 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.152+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc41a5ba0 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.153+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc41039d0 con 0x7f3fc4108090 2024-09-21T01:28:57.157 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.154+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f3fb8042070 con 0x7f3fc4108090 2024-09-21T01:28:57.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.155+0000 7f3fc0ff9640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3fac07af30 0x7f3fac07d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:57.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.155+0000 7f3fc37fe640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3fac07af30 0x7f3fac07d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:57.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.155+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7340+0+0 (secure 0 0 0) 0x7f3fb80befa0 con 0x7f3fc4108090 2024-09-21T01:28:57.158 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.155+0000 7f3fc37fe640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3fac07af30 0x7f3fac07d440 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fc410ed70 tx=0x7f3fb400a2d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:57.161 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.160+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f3fb808a070 con 0x7f3fc4108090 2024-09-21T01:28:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.359+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3fc410a830 con 0x7f3fc4108090 2024-09-21T01:28:57.360 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.359+0000 7f3fc0ff9640 1 -- 172.21.15.45:0/1829453205 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v65) ==== 81+0+1009 (secure 0 0 0) 0x7f3fb808ef20 con 0x7f3fc4108090 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6833/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1147989706 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1797066051 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3851449798 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3707884113 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3194963614 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6832/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1334988077 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:57.361 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1726148947 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:57.362 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/308996383 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:57.362 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:57.362 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:57.362 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:57.362 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1437053407 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:57.362 INFO:teuthology.orchestra.run.smithi045.stderr:listed 17 entries 2024-09-21T01:28:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3fac07af30 msgr2=0x7f3fac07d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3fac07af30 0x7f3fac07d440 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3fc410ed70 tx=0x7f3fb400a2d0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 msgr2=0x7f3fc410d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc410d860 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3fb802a9c0 tx=0x7f3fb80045e0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 shutdown_connections 2024-09-21T01:28:57.364 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f3fac07af30 0x7f3fac07d440 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3fc4108090 0x7f3fc410d860 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 --2- 172.21.15.45:0/1829453205 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fc4102750 0x7f3fc410d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 >> 172.21.15.45:0/1829453205 conn(0x7f3fc40fe1a0 msgr2=0x7f3fc41005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 shutdown_connections 2024-09-21T01:28:57.365 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:57.362+0000 7f3fca1cd640 1 -- 172.21.15.45:0/1829453205 wait complete. 2024-09-21T01:28:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:57 smithi045 ceph-mon[28361]: pgmap v235: 129 pgs: 129 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2024-09-21T01:28:57.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:57 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1829453205' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:28:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:57 smithi090 ceph-mon[35142]: pgmap v235: 129 pgs: 129 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2024-09-21T01:28:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:57 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1829453205' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:28:58.114 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:28:58.114 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:28:58.134 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:28:58.439 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 -- 172.21.15.45:0/2553595579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 msgr2=0x7f4230104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 --2- 172.21.15.45:0/2553595579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230104940 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f422c009c20 tx=0x7f422c02a970 comp rx=0 tx=0).stop 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 -- 172.21.15.45:0/2553595579 shutdown_connections 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 --2- 172.21.15.45:0/2553595579 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 0x7f423010d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 --2- 172.21.15.45:0/2553595579 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230104940 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 -- 172.21.15.45:0/2553595579 >> 172.21.15.45:0/2553595579 conn(0x7f42300fe1a0 msgr2=0x7f42301005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 -- 172.21.15.45:0/2553595579 shutdown_connections 2024-09-21T01:28:58.900 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.898+0000 7f4237b0a640 1 -- 172.21.15.45:0/2553595579 wait complete. 2024-09-21T01:28:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.899+0000 7f4237b0a640 1 Processor -- start 2024-09-21T01:28:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.899+0000 7f4237b0a640 1 -- start start 2024-09-21T01:28:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f4237b0a640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f4237b0a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 0x7f4230199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f4237b0a640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f4230110910 con 0x7f4230104530 2024-09-21T01:28:58.902 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f4237b0a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f4230110790 con 0x7f4230104f60 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:33480/0 (socket says 172.21.15.45:33480) 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 -- 172.21.15.45:0/4207546374 learned_addr learned my addr 172.21.15.45:0/4207546374 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42350a5640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 0x7f4230199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 -- 172.21.15.45:0/4207546374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 msgr2=0x7f4230199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 0x7f4230199f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 -- 172.21.15.45:0/4207546374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423019f1a0 con 0x7f4230104530 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42350a5640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 0x7f4230199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.900+0000 7f42358a6640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230199960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f422c005cc0 tx=0x7f422c0309e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.901+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f422c047070 con 0x7f4230104530 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.901+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f422c02fde0 con 0x7f4230104530 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.901+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f422c034070 con 0x7f4230104530 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.901+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423019f430 con 0x7f4230104530 2024-09-21T01:28:58.903 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.901+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423019f960 con 0x7f4230104530 2024-09-21T01:28:58.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.902+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f423006b230 con 0x7f4230104530 2024-09-21T01:28:58.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.903+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f422c04a070 con 0x7f4230104530 2024-09-21T01:28:58.904 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.903+0000 7f4226ffd640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f421c07af80 0x7f421c07d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:28:58.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.903+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7340+0+0 (secure 0 0 0) 0x7f422c0bf130 con 0x7f4230104530 2024-09-21T01:28:58.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.903+0000 7f42350a5640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f421c07af80 0x7f421c07d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:28:58.905 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.904+0000 7f42350a5640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f421c07af80 0x7f421c07d490 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4230100680 tx=0x7f4220005f90 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:28:58.909 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:58.908+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f422c03f070 con 0x7f4230104530 2024-09-21T01:28:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.112+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4230104940 con 0x7f4230104530 2024-09-21T01:28:59.113 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.112+0000 7f4226ffd640 1 -- 172.21.15.45:0/4207546374 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v65) ==== 81+0+1009 (secure 0 0 0) 0x7f422c08a290 con 0x7f4230104530 2024-09-21T01:28:59.113 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6833/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1147989706 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1797066051 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3851449798 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3707884113 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3194963614 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6832/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1334988077 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1726148947 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/308996383 2024-09-22T01:22:51.944078+0000 2024-09-21T01:28:59.114 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:28:59.115 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:59.115 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:59.115 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1437053407 2024-09-22T01:21:45.172023+0000 2024-09-21T01:28:59.115 INFO:teuthology.orchestra.run.smithi045.stderr:listed 17 entries 2024-09-21T01:28:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f421c07af80 msgr2=0x7f421c07d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:59.116 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f421c07af80 0x7f421c07d490 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4230100680 tx=0x7f4220005f90 comp rx=0 tx=0).stop 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 msgr2=0x7f4230199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230199960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f422c005cc0 tx=0x7f422c0309e0 comp rx=0 tx=0).stop 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 shutdown_connections 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f421c07af80 0x7f421c07d490 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4230104f60 0x7f4230199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 --2- 172.21.15.45:0/4207546374 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4230104530 0x7f4230199960 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 >> 172.21.15.45:0/4207546374 conn(0x7f42300fe1a0 msgr2=0x7f423010cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 shutdown_connections 2024-09-21T01:28:59.117 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:28:59.115+0000 7f4237b0a640 1 -- 172.21.15.45:0/4207546374 wait complete. 2024-09-21T01:28:59.792 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:59 smithi045 ceph-mon[28361]: pgmap v236: 129 pgs: 129 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2024-09-21T01:28:59.792 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:28:59 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4207546374' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:28:59.826 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi045.front.sepia.ceph.com... 2024-09-21T01:28:59.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:28:59.826 DEBUG:teuthology.orchestra.run.smithi045:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-21T01:28:59.856 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi045.stderr:ceph-fuse[79628]: fuse finished with error 0 and tester_r 0 2024-09-21T01:28:59.857 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T01:28:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:59 smithi090 ceph-mon[35142]: pgmap v236: 129 pgs: 129 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2024-09-21T01:28:59.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:28:59 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4207546374' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:29:01.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:01 smithi045 ceph-mon[28361]: pgmap v237: 129 pgs: 129 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2024-09-21T01:29:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:01 smithi090 ceph-mon[35142]: pgmap v237: 129 pgs: 129 active+clean; 582 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s; 72 B/s, 0 objects/s recovering 2024-09-21T01:29:03.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:03 smithi045 ceph-mon[28361]: pgmap v238: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 434 B/s wr, 0 op/s; 61 B/s, 0 objects/s recovering 2024-09-21T01:29:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:03 smithi090 ceph-mon[35142]: pgmap v238: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 434 B/s wr, 0 op/s; 61 B/s, 0 objects/s recovering 2024-09-21T01:29:05.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:05 smithi045 ceph-mon[28361]: pgmap v239: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 392 B/s wr, 0 op/s; 55 B/s, 0 objects/s recovering 2024-09-21T01:29:05.959 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi045.front.sepia.ceph.com 2024-09-21T01:29:05.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:05.959 DEBUG:teuthology.orchestra.run.smithi045:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-21T01:29:05.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:05.991 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-09-21T01:29:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:05 smithi090 ceph-mon[35142]: pgmap v239: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 392 B/s wr, 0 op/s; 55 B/s, 0 objects/s recovering 2024-09-21T01:29:06.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:06.011 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-09-21T01:29:06.086 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-09-21T01:29:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:29:06.087 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-21T01:29:06.087 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-21T01:29:06.087 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph fs ls 2024-09-21T01:29:06.363 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:29:06.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 -- 172.21.15.45:0/150510968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e01091c0 msgr2=0x7fa7e01095d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:06.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 --2- 172.21.15.45:0/150510968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e01091c0 0x7fa7e01095d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009880 tx=0x7fa7d402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:29:06.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 -- 172.21.15.45:0/150510968 shutdown_connections 2024-09-21T01:29:06.831 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 --2- 172.21.15.45:0/150510968 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e01091c0 0x7fa7e01095d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:06.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 --2- 172.21.15.45:0/150510968 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7e0101c80 0x7fa7e0102090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:06.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 -- 172.21.15.45:0/150510968 >> 172.21.15.45:0/150510968 conn(0x7fa7e0079bd0 msgr2=0x7fa7e0079fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:06.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.829+0000 7fa7e51da640 1 -- 172.21.15.45:0/150510968 shutdown_connections 2024-09-21T01:29:06.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.830+0000 7fa7e51da640 1 -- 172.21.15.45:0/150510968 wait complete. 2024-09-21T01:29:06.832 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.830+0000 7fa7e51da640 1 Processor -- start 2024-09-21T01:29:06.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7e51da640 1 -- start start 2024-09-21T01:29:06.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7e51da640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 0x7fa7e01998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:06.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7e51da640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7e01091c0 0x7fa7e0199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:06.833 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7e51da640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0110910 con 0x7fa7e0101c80 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7e51da640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7fa7e0110790 con 0x7fa7e01091c0 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7ded9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 0x7fa7e01998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7ded9d640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 0x7fa7e01998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:54396/0 (socket says 172.21.15.45:54396) 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7ded9d640 1 -- 172.21.15.45:0/1961668585 learned_addr learned my addr 172.21.15.45:0/1961668585 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7de59c640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7e01091c0 0x7fa7e0199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7ded9d640 1 -- 172.21.15.45:0/1961668585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7e01091c0 msgr2=0x7fa7e0199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7ded9d640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7e01091c0 0x7fa7e0199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.831+0000 7fa7ded9d640 1 -- 172.21.15.45:0/1961668585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7e019f120 con 0x7fa7e0101c80 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.832+0000 7fa7ded9d640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 0x7fa7e01998e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000c8f0 tx=0x7fa7d000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.832+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa7d00079b0 con 0x7fa7e0101c80 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.832+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7e019f410 con 0x7fa7e0101c80 2024-09-21T01:29:06.834 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.832+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7fa7d0005020 con 0x7fa7e0101c80 2024-09-21T01:29:06.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.832+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7fa7d0004c50 con 0x7fa7e0101c80 2024-09-21T01:29:06.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.832+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7e019f9f0 con 0x7fa7e0101c80 2024-09-21T01:29:06.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.833+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7e006b4c0 con 0x7fa7e0101c80 2024-09-21T01:29:06.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.834+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7fa7d000f450 con 0x7fa7e0101c80 2024-09-21T01:29:06.835 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.834+0000 7fa7c7fff640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa7c807af30 0x7fa7c807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:29:06.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.834+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7340+0+0 (secure 0 0 0) 0x7fa7d009e3d0 con 0x7fa7e0101c80 2024-09-21T01:29:06.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.834+0000 7fa7de59c640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa7c807af30 0x7fa7c807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:29:06.836 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.835+0000 7fa7de59c640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa7c807af30 0x7fa7c807d440 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009230 tx=0x7fa7d40058d0 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:29:06.840 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:06.839+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7fa7d0014030 con 0x7fa7e0101c80 2024-09-21T01:29:07.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.068+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa7e019a560 con 0x7fa7e0101c80 2024-09-21T01:29:07.069 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.068+0000 7fa7c7fff640 1 -- 172.21.15.45:0/1961668585 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+148 (secure 0 0 0) 0x7fa7d00694a0 con 0x7fa7e0101c80 2024-09-21T01:29:07.070 INFO:teuthology.orchestra.run.smithi045.stdout:name: foo, metadata pool: cephfs.foo.meta, data pools: [cephfs.foo.data ] 2024-09-21T01:29:07.070 INFO:teuthology.orchestra.run.smithi045.stdout:name: bar, metadata pool: cephfs.bar.meta, data pools: [cephfs.bar.data ] 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.071+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa7c807af30 msgr2=0x7fa7c807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.071+0000 7fa7e51da640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa7c807af30 0x7fa7c807d440 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa7d4009230 tx=0x7fa7d40058d0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.071+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 msgr2=0x7fa7e01998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.071+0000 7fa7e51da640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 0x7fa7e01998e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fa7d000c8f0 tx=0x7fa7d000cdb0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 shutdown_connections 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7fa7c807af30 0x7fa7c807d440 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa7e01091c0 0x7fa7e0199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 --2- 172.21.15.45:0/1961668585 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7e0101c80 0x7fa7e01998e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 >> 172.21.15.45:0/1961668585 conn(0x7fa7e0079bd0 msgr2=0x7fa7e0105230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 shutdown_connections 2024-09-21T01:29:07.073 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:29:07.072+0000 7fa7e51da640 1 -- 172.21.15.45:0/1961668585 wait complete. 2024-09-21T01:29:07.738 INFO:tasks.cephfs.mount:Mounting Ceph FS bar; just confirmed its presence on cluster 2024-09-21T01:29:07.738 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-21T01:29:07.738 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-09-21T01:29:07.738 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-21T01:29:07.738 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:07.738 INFO:tasks.cephfs.mount:self.cephfs_name = bar 2024-09-21T01:29:07.739 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-21T01:29:07.739 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-21T01:29:07.739 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-21T01:29:07.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:07.739 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-09-21T01:29:07.758 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-21T01:29:07.758 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-21T01:29:07.758 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-09-21T01:29:07.758 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:29:07.758 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-21T01:29:07.759 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42248sec preferred_lft 42248sec 2024-09-21T01:29:07.760 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link noprefixroute 2024-09-21T01:29:07.760 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-09-21T01:29:07.760 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-21T01:29:07.760 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-21T01:29:07.855 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:29:07.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:07 smithi045 ceph-mon[28361]: pgmap v240: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s; 48 B/s, 0 objects/s recovering 2024-09-21T01:29:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:07 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/1961668585' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:29:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:07.935 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:07 smithi090 ceph-mon[35142]: pgmap v240: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s; 48 B/s, 0 objects/s recovering 2024-09-21T01:29:07.935 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:07 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/1961668585' entity='client.admin' cmd={"prefix": "fs ls"} : dispatch 2024-09-21T01:29:07.935 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:07.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:29:07.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:07.990 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-21T01:29:08.016 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-09-21T01:29:08.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:08.018 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-09-21T01:29:08.086 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-09-21T01:29:08.086 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-21T01:29:08.086 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-21T01:29:08.086 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-21T01:29:08.086 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-21T01:29:08.086 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:29:08.086 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-21T01:29:08.086 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-21T01:29:08.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-21T01:29:08.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-21T01:29:08.087 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-21T01:29:08.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:29:08.373 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:29:08.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:08.379 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-09-21T01:29:08.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:08.394 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-09-21T01:29:08.449 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:29:08.449 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-21T01:29:08.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-21T01:29:08.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-21T01:29:08.450 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-21T01:29:08.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:29:08.560 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:29:08.566 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-21T01:29:08.566 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:29:08.566 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-21T01:29:08.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-21T01:29:08.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-21T01:29:08.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-21T01:29:08.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-21T01:29:08.567 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-21T01:29:08.567 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-21T01:29:08.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:29:08.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:29:08.689 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-09-21T01:29:08.689 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-21T01:29:08.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-09-21T01:29:08.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-09-21T01:29:08.689 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-21T01:29:08.737 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-09-21T01:29:08.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:29:08.764 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'bar'} 2024-09-21T01:29:08.764 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:29:08.765 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:08.779 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-21T01:29:08.779 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-21T01:29:08.779 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:08.835 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-09-21T01:29:08.900 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/0 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/869cfd88cd0a2187cd49fb3a70e1d7f4e7c5f236f0b74323d7947f16bbf365bb/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/ed691fee861fb4b0bde2290e235eb9ab04f56af54c4c74d33d05e63e06ce8a15/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/deb903897ed3d2f02eeb0a4481a0d6ca24b1aa863f1e5d35932815543fbb36b3/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/73d51a598bcbb34abeaacf559fbd4c56d390e01bd2e6a713fe5c635973529c8b/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/eb444863c7d36b2a1be5bb3d4ec4c71048315af14975b0caf7fbd1f34e689ee8/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/1001c2536eba26c0e2a68bb3a42d1916612c1a7ddfd54fd570ee2e524bed3c21/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/f2d221d03bfec694dbb85ce9df86ad06b197d23f593613e2d7336e7bc1064d31/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/c58bf6345d9ff07bcedacfc9f3922e4da078802cffd44662c56a52a0444081e5/merged 2024-09-21T01:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/b57b75d48932822547ff496ce633a8a2fd848197dcba502c21248e9f1414770c/merged 2024-09-21T01:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/7f99526ef679f9c1b0bea494069e8c2971a3eb4928fdff7c0b8025ddfff781e2/merged 2024-09-21T01:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/0bba2b96ed53110d3f48cc955142c686685a21d53e430dc9949fe7213450a720/merged 2024-09-21T01:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-09-21T01:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-21T01:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-21T01:29:08.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:08.960 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-09-21T01:29:09.012 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-21T01:29:09.012 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=bar) 2024-09-21T01:29:09.055 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-09-21T01:29:09.079 DEBUG:teuthology.orchestra.run.smithi090:> cat /proc/self/mounts | awk '{print $2}' 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/proc 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/sys 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/dev 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/security 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/shm 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/pts 2024-09-21T01:29:09.136 INFO:teuthology.orchestra.run.smithi090.stdout:/run 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/cgroup 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/pstore 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/bpf 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/ 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/selinux 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/debug 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/mqueue 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/dev/hugepages 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/tracing 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/fs/fuse/connections 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/sys/kernel/config 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-sysctl.service 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-09-21T01:29:09.137 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/nfs/rpc_pipefs 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/1000 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/ceph 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/run/user/0 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/proc/sys/fs/binfmt_misc 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/869cfd88cd0a2187cd49fb3a70e1d7f4e7c5f236f0b74323d7947f16bbf365bb/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/ed691fee861fb4b0bde2290e235eb9ab04f56af54c4c74d33d05e63e06ce8a15/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/deb903897ed3d2f02eeb0a4481a0d6ca24b1aa863f1e5d35932815543fbb36b3/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/73d51a598bcbb34abeaacf559fbd4c56d390e01bd2e6a713fe5c635973529c8b/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/eb444863c7d36b2a1be5bb3d4ec4c71048315af14975b0caf7fbd1f34e689ee8/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/1001c2536eba26c0e2a68bb3a42d1916612c1a7ddfd54fd570ee2e524bed3c21/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/f2d221d03bfec694dbb85ce9df86ad06b197d23f593613e2d7336e7bc1064d31/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/c58bf6345d9ff07bcedacfc9f3922e4da078802cffd44662c56a52a0444081e5/merged 2024-09-21T01:29:09.138 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/b57b75d48932822547ff496ce633a8a2fd848197dcba502c21248e9f1414770c/merged 2024-09-21T01:29:09.139 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/7f99526ef679f9c1b0bea494069e8c2971a3eb4928fdff7c0b8025ddfff781e2/merged 2024-09-21T01:29:09.139 INFO:teuthology.orchestra.run.smithi090.stdout:/var/lib/containers/storage/overlay/0bba2b96ed53110d3f48cc955142c686685a21d53e430dc9949fe7213450a720/merged 2024-09-21T01:29:09.139 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns 2024-09-21T01:29:09.139 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-21T01:29:09.139 INFO:teuthology.orchestra.run.smithi090.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-21T01:29:09.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:09.139 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-09-21T01:29:09.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi090.stderr:2024-09-21T01:29:09.180+0000 7fbc7e362480 -1 init, newargv = 0x7fbc6c056e90 newargc=15 2024-09-21T01:29:09.180 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi090.stderr:ceph-fuse[61750]: starting ceph client 2024-09-21T01:29:09.187 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi090.stderr:ceph-fuse[61750]: starting fuse 2024-09-21T01:29:09.192 INFO:teuthology.orchestra.run.smithi090.stdout:132 2024-09-21T01:29:09.192 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [132] 2024-09-21T01:29:09.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo stdin-killer -- python3 -c ' 2024-09-21T01:29:09.192 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-09-21T01:29:09.192 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-09-21T01:29:09.192 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-09-21T01:29:09.192 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-09-21T01:29:09.192 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-09-21T01:29:09.193 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-21T01:29:09.194 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-21T01:29:09.194 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.1")) 2024-09-21T01:29:09.194 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-09-21T01:29:09.309 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.1.61750.asok 2024-09-21T01:29:09.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-21T01:29:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-09-21T01:29:09.319 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.61750.asok 2024-09-21T01:29:09.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:09.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.61750.asok status 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "1843221895adccfb3b692060251dd6a0183bc410", 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 19.3.0-5120-g18432218 (1843221895adccfb3b692060251dd6a0183bc410) squid (dev)", 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "1", 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-09-21T01:29:09.461 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "61750", 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24439, 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 24439 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-09-21T01:29:09.462 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3436574518 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 3436574518 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.24439 192.168.0.1:0/3436574518", 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/3436574518", 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 15, 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 65, 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-09-21T01:29:09.463 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-09-21T01:29:09.464 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "bar" 2024-09-21T01:29:09.464 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-09-21T01:29:09.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:09.470 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-09-21T01:29:09.470 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:09.485 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-09-21T01:29:09.485 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:09.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:29:09.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:09.551 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T01:29:09.565 INFO:tasks.workunit:Pulling workunits from ref 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:29:09.565 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T01:29:09.565 INFO:tasks.workunit:timeout=3h 2024-09-21T01:29:09.565 INFO:tasks.workunit:cleanup=True 2024-09-21T01:29:09.566 DEBUG:teuthology.orchestra.run.smithi045:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T01:29:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T01:29:09.587 INFO:teuthology.orchestra.run.smithi045.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T01:29:09.587 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T01:29:09.646 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T01:29:09.646 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T01:29:09.706 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 84h/132d Inode: 1 Links: 2 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:fusefs_t:s0 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-21 01:28:28.825226398 +0000 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-21 01:29:09.549456736 +0000 2024-09-21T01:29:09.724 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2024-09-21T01:29:09.725 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-21T01:29:09.725 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-21T01:29:09.797 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:09 smithi090 ceph-mon[35142]: pgmap v241: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T01:29:09.798 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:29:09.798 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 1843221895adccfb3b692060251dd6a0183bc410 2024-09-21T01:29:09.818 INFO:tasks.workunit.client.0.smithi045.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T01:29:09.855 INFO:tasks.workunit.client.1.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-21T01:29:09.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:09 smithi045 ceph-mon[28361]: pgmap v241: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T01:29:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:11 smithi045 ceph-mon[28361]: pgmap v242: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T01:29:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:11 smithi090 ceph-mon[35142]: pgmap v242: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T01:29:13.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:13 smithi045 ceph-mon[28361]: pgmap v243: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T01:29:13.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:13 smithi090 ceph-mon[35142]: pgmap v243: 129 pgs: 129 active+clean; 583 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T01:29:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:15 smithi045 ceph-mon[28361]: pgmap v244: 129 pgs: 129 active+clean; 586 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T01:29:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:15 smithi090 ceph-mon[35142]: pgmap v244: 129 pgs: 129 active+clean; 586 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T01:29:17.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:17 smithi045 ceph-mon[28361]: pgmap v245: 129 pgs: 129 active+clean; 586 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 597 B/s wr, 0 op/s 2024-09-21T01:29:17.989 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:17 smithi090 ceph-mon[35142]: pgmap v245: 129 pgs: 129 active+clean; 586 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 597 B/s wr, 0 op/s 2024-09-21T01:29:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:19 smithi045 ceph-mon[28361]: pgmap v246: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-09-21T01:29:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:19 smithi090 ceph-mon[35142]: pgmap v246: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-09-21T01:29:21.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:21 smithi045 ceph-mon[28361]: pgmap v247: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-09-21T01:29:21.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:21 smithi090 ceph-mon[35142]: pgmap v247: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-09-21T01:29:22.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:23.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:23 smithi045 ceph-mon[28361]: pgmap v248: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-09-21T01:29:23.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:23 smithi090 ceph-mon[35142]: pgmap v248: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-09-21T01:29:25.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:25 smithi045 ceph-mon[28361]: pgmap v249: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s 2024-09-21T01:29:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:25 smithi090 ceph-mon[35142]: pgmap v249: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 682 B/s wr, 0 op/s 2024-09-21T01:29:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:27 smithi045 ceph-mon[28361]: pgmap v250: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T01:29:27.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:27 smithi090 ceph-mon[35142]: pgmap v250: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T01:29:29.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:29 smithi045 ceph-mon[28361]: pgmap v251: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T01:29:29.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:29 smithi090 ceph-mon[35142]: pgmap v251: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T01:29:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:31 smithi090 ceph-mon[35142]: pgmap v252: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:31 smithi045 ceph-mon[28361]: pgmap v252: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:33 smithi090 ceph-mon[35142]: pgmap v253: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:33 smithi045 ceph-mon[28361]: pgmap v253: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:35.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:35 smithi090 ceph-mon[35142]: pgmap v254: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:35 smithi045 ceph-mon[28361]: pgmap v254: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:37 smithi090 ceph-mon[35142]: pgmap v255: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:37 smithi045 ceph-mon[28361]: pgmap v255: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:39.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:39 smithi045 ceph-mon[28361]: pgmap v256: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:39.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:39 smithi090 ceph-mon[35142]: pgmap v256: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:41 smithi090 ceph-mon[35142]: pgmap v257: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:41 smithi045 ceph-mon[28361]: pgmap v257: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:43.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:43 smithi090 ceph-mon[35142]: pgmap v258: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:43 smithi045 ceph-mon[28361]: pgmap v258: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:45 smithi090 ceph-mon[35142]: pgmap v259: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:45 smithi045 ceph-mon[28361]: pgmap v259: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:47 smithi090 ceph-mon[35142]: pgmap v260: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:47 smithi045 ceph-mon[28361]: pgmap v260: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:49 smithi090 ceph-mon[35142]: pgmap v261: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:50.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:49 smithi045 ceph-mon[28361]: pgmap v261: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:50.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:50 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:29:50.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:50 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:29:50.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:50 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:29:50.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:50 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:29:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:50 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:29:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:50 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:29:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:50 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:29:51.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:50 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:29:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:51 smithi090 ceph-mon[35142]: pgmap v262: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:51 smithi045 ceph-mon[28361]: pgmap v262: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:29:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:53 smithi090 ceph-mon[35142]: pgmap v263: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:54.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:53 smithi045 ceph-mon[28361]: pgmap v263: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:55 smithi090 ceph-mon[35142]: pgmap v264: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:55 smithi045 ceph-mon[28361]: pgmap v264: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:57.971 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:57 smithi045 ceph-mon[28361]: pgmap v265: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:57 smithi090 ceph-mon[35142]: pgmap v265: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:29:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:29:59 smithi090 ceph-mon[35142]: pgmap v266: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:00.104 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:29:59 smithi045 ceph-mon[28361]: pgmap v266: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:00.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:00 smithi045 ceph-mon[28361]: overall HEALTH_OK 2024-09-21T01:30:00.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:00 smithi090 ceph-mon[35142]: overall HEALTH_OK 2024-09-21T01:30:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:01 smithi090 ceph-mon[35142]: pgmap v267: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:01 smithi045 ceph-mon[28361]: pgmap v267: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:03 smithi090 ceph-mon[35142]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:03 smithi045 ceph-mon[28361]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:05 smithi090 ceph-mon[35142]: pgmap v269: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:05 smithi045 ceph-mon[28361]: pgmap v269: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:07 smithi090 ceph-mon[35142]: pgmap v270: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:07 smithi045 ceph-mon[28361]: pgmap v270: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:09.987 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:09 smithi045 ceph-mon[28361]: pgmap v271: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:09.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:09 smithi090 ceph-mon[35142]: pgmap v271: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:10.218 INFO:tasks.workunit.client.1.smithi090.stderr:Note: switching to '1843221895adccfb3b692060251dd6a0183bc410'. 2024-09-21T01:30:10.218 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: git switch -c 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:Or undo this operation with: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: git switch - 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr: 2024-09-21T01:30:10.219 INFO:tasks.workunit.client.1.smithi090.stderr:HEAD is now at 1843221895a Merge PR #56776 into wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:30:10.224 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-21T01:30:10.292 INFO:tasks.workunit.client.1.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T01:30:10.294 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-21T01:30:10.294 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T01:30:11.076 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T01:30:11.172 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T01:30:11.213 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-21T01:30:11.214 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-21T01:30:11.214 INFO:tasks.workunit.client.1.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T01:30:11.253 INFO:tasks.workunit.client.1.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-21T01:30:11.256 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:30:11.256 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-21T01:30:11.312 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.1... 2024-09-21T01:30:11.314 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-21T01:30:11.314 DEBUG:teuthology.orchestra.run.smithi090:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1843221895adccfb3b692060251dd6a0183bc410 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh 2024-09-21T01:30:11.378 INFO:tasks.workunit.client.1.smithi090.stderr:+ dbench 1 2024-09-21T01:30:11.388 INFO:tasks.workunit.client.1.smithi090.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T01:30:11.388 INFO:tasks.workunit.client.1.smithi090.stdout: 2024-09-21T01:30:11.388 INFO:tasks.workunit.client.1.smithi090.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T01:30:11.393 INFO:tasks.workunit.client.1.smithi090.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-21T01:30:11.415 INFO:tasks.workunit.client.0.smithi045.stderr:Updating files: 93% (11823/12697) Updating files: 94% (11936/12697) Updating files: 95% (12063/12697) Updating files: 96% (12190/12697) Updating files: 97% (12317/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-21T01:30:11.493 INFO:tasks.workunit.client.1.smithi090.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-21T01:30:11.493 INFO:tasks.workunit.client.1.smithi090.stdout:releasing clients 2024-09-21T01:30:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:11 smithi090 ceph-mon[35142]: pgmap v272: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:11 smithi045 ceph-mon[28361]: pgmap v272: 129 pgs: 129 active+clean; 587 KiB data, 222 MiB used, 715 GiB / 715 GiB avail 2024-09-21T01:30:12.216 INFO:tasks.workunit.client.0.smithi045.stderr:Note: switching to '1843221895adccfb3b692060251dd6a0183bc410'. 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: git switch -c 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr:Or undo this operation with: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.217 INFO:tasks.workunit.client.0.smithi045.stderr: git switch - 2024-09-21T01:30:12.218 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.218 INFO:tasks.workunit.client.0.smithi045.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T01:30:12.218 INFO:tasks.workunit.client.0.smithi045.stderr: 2024-09-21T01:30:12.218 INFO:tasks.workunit.client.0.smithi045.stderr:HEAD is now at 1843221895a Merge PR #56776 into wip-pdonnell-testing-20240920.212106-debug 2024-09-21T01:30:12.224 DEBUG:teuthology.orchestra.run.smithi045:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T01:30:12.293 INFO:tasks.workunit.client.0.smithi045.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T01:30:12.296 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T01:30:12.296 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T01:30:12.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 980 21.90 MB/sec warmup 1 sec latency 13.143 ms 2024-09-21T01:30:13.029 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T01:30:13.223 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T01:30:13.268 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T01:30:13.269 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T01:30:13.269 INFO:tasks.workunit.client.0.smithi045.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T01:30:13.311 INFO:tasks.workunit.client.0.smithi045.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T01:30:13.315 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:30:13.315 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T01:30:13.372 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-09-21T01:30:13.374 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-21T01:30:13.374 DEBUG:teuthology.orchestra.run.smithi045:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1843221895adccfb3b692060251dd6a0183bc410 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-09-21T01:30:13.433 INFO:tasks.workunit.client.0.smithi045.stderr:+ dbench 1 2024-09-21T01:30:13.441 INFO:tasks.workunit.client.0.smithi045.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T01:30:13.442 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-09-21T01:30:13.442 INFO:tasks.workunit.client.0.smithi045.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T01:30:13.451 INFO:tasks.workunit.client.0.smithi045.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-21T01:30:13.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 1820 13.17 MB/sec warmup 2 sec latency 13.523 ms 2024-09-21T01:30:13.551 INFO:tasks.workunit.client.0.smithi045.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-21T01:30:13.552 INFO:tasks.workunit.client.0.smithi045.stdout:releasing clients 2024-09-21T01:30:13.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:13 smithi090 ceph-mon[35142]: pgmap v273: 129 pgs: 129 active+clean; 2.6 MiB data, 222 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s wr, 0 op/s 2024-09-21T01:30:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:13 smithi045 ceph-mon[28361]: pgmap v273: 129 pgs: 129 active+clean; 2.6 MiB data, 222 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s wr, 0 op/s 2024-09-21T01:30:14.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 2561 9.64 MB/sec warmup 3 sec latency 14.022 ms 2024-09-21T01:30:14.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 2768 31.97 MB/sec warmup 1 sec latency 113.703 ms 2024-09-21T01:30:15.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 3359 8.86 MB/sec warmup 4 sec latency 12.783 ms 2024-09-21T01:30:15.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 5453 22.75 MB/sec warmup 2 sec latency 38.605 ms 2024-09-21T01:30:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:15 smithi090 ceph-mon[35142]: pgmap v274: 129 pgs: 129 active+clean; 9.3 MiB data, 231 MiB used, 715 GiB / 715 GiB avail; 743 KiB/s wr, 6 op/s 2024-09-21T01:30:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:15 smithi045 ceph-mon[28361]: pgmap v274: 129 pgs: 129 active+clean; 9.3 MiB data, 231 MiB used, 715 GiB / 715 GiB avail; 743 KiB/s wr, 6 op/s 2024-09-21T01:30:16.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 4028 8.01 MB/sec warmup 5 sec latency 13.159 ms 2024-09-21T01:30:16.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 8740 21.47 MB/sec warmup 3 sec latency 41.622 ms 2024-09-21T01:30:17.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 4656 7.01 MB/sec warmup 6 sec latency 49.617 ms 2024-09-21T01:30:17.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 11388 19.67 MB/sec warmup 4 sec latency 239.605 ms 2024-09-21T01:30:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:17 smithi090 ceph-mon[35142]: pgmap v275: 129 pgs: 129 active+clean; 21 MiB data, 271 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-21T01:30:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:17 smithi045 ceph-mon[28361]: pgmap v275: 129 pgs: 129 active+clean; 21 MiB data, 271 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 1.8 MiB/s wr, 18 op/s 2024-09-21T01:30:18.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 5450 6.50 MB/sec warmup 7 sec latency 20.301 ms 2024-09-21T01:30:18.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 14884 19.53 MB/sec warmup 5 sec latency 49.949 ms 2024-09-21T01:30:19.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 6275 6.38 MB/sec warmup 8 sec latency 14.091 ms 2024-09-21T01:30:19.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 18420 19.46 MB/sec warmup 6 sec latency 66.297 ms 2024-09-21T01:30:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:19 smithi090 ceph-mon[35142]: pgmap v276: 129 pgs: 129 active+clean; 25 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.3 MiB/s wr, 42 op/s 2024-09-21T01:30:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:19 smithi045 ceph-mon[28361]: pgmap v276: 129 pgs: 129 active+clean; 25 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.3 MiB/s wr, 42 op/s 2024-09-21T01:30:20.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 7084 6.39 MB/sec warmup 9 sec latency 14.455 ms 2024-09-21T01:30:20.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 21084 18.72 MB/sec warmup 7 sec latency 35.134 ms 2024-09-21T01:30:21.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 7663 5.92 MB/sec warmup 10 sec latency 9.855 ms 2024-09-21T01:30:21.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 24305 18.54 MB/sec warmup 8 sec latency 33.160 ms 2024-09-21T01:30:21.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:21 smithi090 ceph-mon[35142]: pgmap v277: 129 pgs: 129 active+clean; 25 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.4 MiB/s wr, 45 op/s 2024-09-21T01:30:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:21 smithi045 ceph-mon[28361]: pgmap v277: 129 pgs: 129 active+clean; 25 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.4 MiB/s wr, 45 op/s 2024-09-21T01:30:22.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 8359 5.58 MB/sec warmup 11 sec latency 21.971 ms 2024-09-21T01:30:22.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 26363 17.77 MB/sec warmup 9 sec latency 125.056 ms 2024-09-21T01:30:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:23.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 9159 5.41 MB/sec warmup 12 sec latency 54.184 ms 2024-09-21T01:30:23.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 26765 16.03 MB/sec warmup 10 sec latency 315.754 ms 2024-09-21T01:30:24.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:23 smithi045 ceph-mon[28361]: pgmap v278: 129 pgs: 129 active+clean; 25 MiB data, 298 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.7 MiB/s wr, 57 op/s 2024-09-21T01:30:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:23 smithi090 ceph-mon[35142]: pgmap v278: 129 pgs: 129 active+clean; 25 MiB data, 298 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.7 MiB/s wr, 57 op/s 2024-09-21T01:30:24.493 INFO:tasks.workunit.client.1.smithi090.stdout: 1 9979 5.42 MB/sec warmup 13 sec latency 57.956 ms 2024-09-21T01:30:24.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 27252 14.71 MB/sec warmup 11 sec latency 280.074 ms 2024-09-21T01:30:25.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 10629 5.47 MB/sec warmup 14 sec latency 22.188 ms 2024-09-21T01:30:25.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 27793 13.95 MB/sec warmup 12 sec latency 316.690 ms 2024-09-21T01:30:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:25 smithi045 ceph-mon[28361]: pgmap v279: 129 pgs: 129 active+clean; 26 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.9 MiB/s wr, 80 op/s 2024-09-21T01:30:26.193 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:25 smithi090 ceph-mon[35142]: pgmap v279: 129 pgs: 129 active+clean; 26 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2.9 MiB/s wr, 80 op/s 2024-09-21T01:30:26.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 11157 5.22 MB/sec warmup 15 sec latency 43.942 ms 2024-09-21T01:30:26.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 29217 13.42 MB/sec warmup 13 sec latency 279.390 ms 2024-09-21T01:30:27.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 11863 5.03 MB/sec warmup 16 sec latency 17.018 ms 2024-09-21T01:30:27.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 32648 13.81 MB/sec warmup 14 sec latency 149.577 ms 2024-09-21T01:30:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:27 smithi045 ceph-mon[28361]: pgmap v280: 129 pgs: 129 active+clean; 26 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 MiB/s wr, 87 op/s 2024-09-21T01:30:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:27 smithi090 ceph-mon[35142]: pgmap v280: 129 pgs: 129 active+clean; 26 MiB data, 309 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 MiB/s wr, 87 op/s 2024-09-21T01:30:28.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 12706 4.94 MB/sec warmup 17 sec latency 12.884 ms 2024-09-21T01:30:28.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 35295 13.83 MB/sec warmup 15 sec latency 49.410 ms 2024-09-21T01:30:29.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 13579 5.05 MB/sec warmup 18 sec latency 14.605 ms 2024-09-21T01:30:29.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 38504 14.05 MB/sec warmup 16 sec latency 30.180 ms 2024-09-21T01:30:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:29 smithi045 ceph-mon[28361]: pgmap v281: 129 pgs: 129 active+clean; 27 MiB data, 313 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-21T01:30:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:29 smithi090 ceph-mon[35142]: pgmap v281: 129 pgs: 129 active+clean; 27 MiB data, 313 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-21T01:30:30.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 14333 5.06 MB/sec warmup 19 sec latency 13.370 ms 2024-09-21T01:30:30.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 41475 14.03 MB/sec warmup 17 sec latency 66.186 ms 2024-09-21T01:30:31.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 14869 4.88 MB/sec warmup 20 sec latency 19.120 ms 2024-09-21T01:30:31.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 43403 13.94 MB/sec warmup 18 sec latency 224.186 ms 2024-09-21T01:30:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:31 smithi045 ceph-mon[28361]: pgmap v282: 129 pgs: 129 active+clean; 28 MiB data, 317 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-21T01:30:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:31 smithi090 ceph-mon[35142]: pgmap v282: 129 pgs: 129 active+clean; 28 MiB data, 317 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-21T01:30:32.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 15623 4.88 MB/sec warmup 21 sec latency 37.291 ms 2024-09-21T01:30:32.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 43481 13.22 MB/sec warmup 19 sec latency 322.259 ms 2024-09-21T01:30:33.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 16456 4.69 MB/sec warmup 22 sec latency 12.199 ms 2024-09-21T01:30:33.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 43587 12.57 MB/sec warmup 20 sec latency 305.838 ms 2024-09-21T01:30:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:33 smithi045 ceph-mon[28361]: pgmap v283: 129 pgs: 129 active+clean; 28 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:30:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:33 smithi090 ceph-mon[35142]: pgmap v283: 129 pgs: 129 active+clean; 28 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:30:34.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 17458 4.86 MB/sec warmup 23 sec latency 12.265 ms 2024-09-21T01:30:34.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 44237 12.17 MB/sec warmup 21 sec latency 228.786 ms 2024-09-21T01:30:35.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 17994 4.84 MB/sec warmup 24 sec latency 13.275 ms 2024-09-21T01:30:35.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 44944 11.70 MB/sec warmup 22 sec latency 293.497 ms 2024-09-21T01:30:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:35 smithi045 ceph-mon[28361]: pgmap v284: 129 pgs: 129 active+clean; 28 MiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:30:36.194 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:35 smithi090 ceph-mon[35142]: pgmap v284: 129 pgs: 129 active+clean; 28 MiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:30:36.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 18533 4.68 MB/sec warmup 25 sec latency 19.339 ms 2024-09-21T01:30:36.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 47785 11.95 MB/sec warmup 23 sec latency 194.992 ms 2024-09-21T01:30:37.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 19305 4.68 MB/sec warmup 26 sec latency 14.104 ms 2024-09-21T01:30:37.552 INFO:tasks.workunit.client.0.smithi045.stdout: 1 50678 12.07 MB/sec warmup 24 sec latency 33.434 ms 2024-09-21T01:30:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:37 smithi045 ceph-mon[28361]: pgmap v285: 129 pgs: 129 active+clean; 28 MiB data, 331 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-21T01:30:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:37 smithi090 ceph-mon[35142]: pgmap v285: 129 pgs: 129 active+clean; 28 MiB data, 331 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-21T01:30:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:38.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 20151 4.58 MB/sec warmup 27 sec latency 17.112 ms 2024-09-21T01:30:38.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 54049 12.33 MB/sec warmup 25 sec latency 57.505 ms 2024-09-21T01:30:39.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 21064 4.68 MB/sec warmup 28 sec latency 71.813 ms 2024-09-21T01:30:39.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 57515 12.59 MB/sec warmup 26 sec latency 108.190 ms 2024-09-21T01:30:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:39 smithi045 ceph-mon[28361]: pgmap v286: 129 pgs: 129 active+clean; 29 MiB data, 339 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:30:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:39 smithi090 ceph-mon[35142]: pgmap v286: 129 pgs: 129 active+clean; 29 MiB data, 339 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:30:40.494 INFO:tasks.workunit.client.1.smithi090.stdout: 1 21542 4.67 MB/sec warmup 29 sec latency 98.746 ms 2024-09-21T01:30:40.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 57522 12.12 MB/sec warmup 27 sec latency 316.903 ms 2024-09-21T01:30:41.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 21990 4.53 MB/sec warmup 30 sec latency 34.364 ms 2024-09-21T01:30:41.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 59776 12.13 MB/sec warmup 28 sec latency 258.476 ms 2024-09-21T01:30:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:41 smithi045 ceph-mon[28361]: pgmap v287: 129 pgs: 129 active+clean; 31 MiB data, 341 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-09-21T01:30:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:41 smithi090 ceph-mon[35142]: pgmap v287: 129 pgs: 129 active+clean; 31 MiB data, 341 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-09-21T01:30:42.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 22793 4.54 MB/sec warmup 31 sec latency 13.803 ms 2024-09-21T01:30:42.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 62225 12.12 MB/sec warmup 29 sec latency 116.958 ms 2024-09-21T01:30:43.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 23647 4.46 MB/sec warmup 32 sec latency 12.858 ms 2024-09-21T01:30:43.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 65510 12.35 MB/sec warmup 30 sec latency 33.425 ms 2024-09-21T01:30:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:43 smithi045 ceph-mon[28361]: pgmap v288: 129 pgs: 129 active+clean; 30 MiB data, 340 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:30:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:43 smithi090 ceph-mon[35142]: pgmap v288: 129 pgs: 129 active+clean; 30 MiB data, 340 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:30:44.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 24605 4.55 MB/sec warmup 33 sec latency 18.068 ms 2024-09-21T01:30:44.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 68413 12.43 MB/sec warmup 31 sec latency 33.230 ms 2024-09-21T01:30:45.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 25158 4.55 MB/sec warmup 34 sec latency 13.278 ms 2024-09-21T01:30:45.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 71775 12.62 MB/sec warmup 32 sec latency 58.304 ms 2024-09-21T01:30:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:45 smithi045 ceph-mon[28361]: pgmap v289: 129 pgs: 129 active+clean; 31 MiB data, 346 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:30:46.194 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:45 smithi090 ceph-mon[35142]: pgmap v289: 129 pgs: 129 active+clean; 31 MiB data, 346 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:30:46.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 25739 4.44 MB/sec warmup 35 sec latency 17.145 ms 2024-09-21T01:30:46.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 73975 12.61 MB/sec warmup 33 sec latency 268.695 ms 2024-09-21T01:30:47.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 26470 4.45 MB/sec warmup 36 sec latency 27.498 ms 2024-09-21T01:30:47.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 74290 12.29 MB/sec warmup 34 sec latency 358.291 ms 2024-09-21T01:30:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:47 smithi045 ceph-mon[28361]: pgmap v290: 129 pgs: 129 active+clean; 33 MiB data, 347 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-09-21T01:30:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:47 smithi090 ceph-mon[35142]: pgmap v290: 129 pgs: 129 active+clean; 33 MiB data, 347 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-09-21T01:30:48.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 27267 4.37 MB/sec warmup 37 sec latency 49.423 ms 2024-09-21T01:30:48.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 77022 12.30 MB/sec warmup 35 sec latency 257.926 ms 2024-09-21T01:30:49.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 28178 4.45 MB/sec warmup 38 sec latency 12.479 ms 2024-09-21T01:30:49.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 80433 12.46 MB/sec warmup 36 sec latency 33.254 ms 2024-09-21T01:30:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:49 smithi045 ceph-mon[28361]: pgmap v291: 129 pgs: 129 active+clean; 32 MiB data, 349 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:30:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:49 smithi090 ceph-mon[35142]: pgmap v291: 129 pgs: 129 active+clean; 32 MiB data, 349 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:30:50.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 28773 4.46 MB/sec warmup 39 sec latency 13.683 ms 2024-09-21T01:30:50.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 83123 12.59 MB/sec warmup 37 sec latency 66.761 ms 2024-09-21T01:30:51.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 29346 4.37 MB/sec warmup 40 sec latency 22.596 ms 2024-09-21T01:30:51.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 86103 12.65 MB/sec warmup 38 sec latency 33.192 ms 2024-09-21T01:30:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:51 smithi045 ceph-mon[28361]: pgmap v292: 129 pgs: 129 active+clean; 32 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T01:30:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:30:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:30:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:30:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:51 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:30:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:51 smithi090 ceph-mon[35142]: pgmap v292: 129 pgs: 129 active+clean; 32 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T01:30:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:51 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:30:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:51 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:30:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:51 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:30:52.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:51 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:30:52.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 30062 4.37 MB/sec warmup 41 sec latency 17.332 ms 2024-09-21T01:30:52.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 89538 12.81 MB/sec warmup 39 sec latency 58.533 ms 2024-09-21T01:30:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:30:53.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 30835 4.31 MB/sec warmup 42 sec latency 15.548 ms 2024-09-21T01:30:53.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 93072 12.97 MB/sec warmup 40 sec latency 32.302 ms 2024-09-21T01:30:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:53 smithi045 ceph-mon[28361]: pgmap v293: 129 pgs: 129 active+clean; 34 MiB data, 360 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:30:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:53 smithi090 ceph-mon[35142]: pgmap v293: 129 pgs: 129 active+clean; 34 MiB data, 360 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:30:54.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 31729 4.38 MB/sec warmup 43 sec latency 14.365 ms 2024-09-21T01:30:54.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 96556 13.11 MB/sec warmup 41 sec latency 32.725 ms 2024-09-21T01:30:55.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 32348 4.39 MB/sec warmup 44 sec latency 12.248 ms 2024-09-21T01:30:55.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 100045 13.25 MB/sec warmup 42 sec latency 49.036 ms 2024-09-21T01:30:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:55 smithi045 ceph-mon[28361]: pgmap v294: 129 pgs: 129 active+clean; 34 MiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:30:56.194 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:55 smithi090 ceph-mon[35142]: pgmap v294: 129 pgs: 129 active+clean; 34 MiB data, 366 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:30:56.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 32944 4.31 MB/sec warmup 45 sec latency 16.927 ms 2024-09-21T01:30:56.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 102668 13.28 MB/sec warmup 43 sec latency 66.609 ms 2024-09-21T01:30:57.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 33792 4.31 MB/sec warmup 46 sec latency 12.412 ms 2024-09-21T01:30:57.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 105829 13.37 MB/sec warmup 44 sec latency 41.917 ms 2024-09-21T01:30:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:57 smithi045 ceph-mon[28361]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 373 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 108 op/s 2024-09-21T01:30:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:57 smithi090 ceph-mon[35142]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 373 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 108 op/s 2024-09-21T01:30:58.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 34523 4.28 MB/sec warmup 47 sec latency 17.852 ms 2024-09-21T01:30:58.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 107440 13.24 MB/sec warmup 45 sec latency 133.552 ms 2024-09-21T01:30:59.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 35302 4.32 MB/sec warmup 48 sec latency 12.471 ms 2024-09-21T01:30:59.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 110887 13.36 MB/sec warmup 46 sec latency 26.965 ms 2024-09-21T01:30:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:30:59 smithi090 ceph-mon[35142]: pgmap v296: 129 pgs: 129 active+clean; 35 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:31:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:30:59 smithi045 ceph-mon[28361]: pgmap v296: 129 pgs: 129 active+clean; 35 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:31:00.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 35965 4.33 MB/sec warmup 49 sec latency 13.360 ms 2024-09-21T01:31:00.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 114344 13.48 MB/sec warmup 47 sec latency 83.502 ms 2024-09-21T01:31:01.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 36650 4.28 MB/sec warmup 50 sec latency 17.204 ms 2024-09-21T01:31:01.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 117824 13.59 MB/sec warmup 48 sec latency 40.956 ms 2024-09-21T01:31:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:01 smithi045 ceph-mon[28361]: pgmap v297: 129 pgs: 129 active+clean; 35 MiB data, 376 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:31:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:01 smithi090 ceph-mon[35142]: pgmap v297: 129 pgs: 129 active+clean; 35 MiB data, 376 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:31:02.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 37486 4.27 MB/sec warmup 51 sec latency 14.881 ms 2024-09-21T01:31:02.553 INFO:tasks.workunit.client.0.smithi045.stdout: 1 121317 13.70 MB/sec warmup 49 sec latency 33.635 ms 2024-09-21T01:31:03.495 INFO:tasks.workunit.client.1.smithi090.stdout: 1 38273 4.29 MB/sec warmup 52 sec latency 11.324 ms 2024-09-21T01:31:03.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 123959 13.71 MB/sec warmup 50 sec latency 33.472 ms 2024-09-21T01:31:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:03 smithi045 ceph-mon[28361]: pgmap v298: 129 pgs: 129 active+clean; 36 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-21T01:31:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:03 smithi090 ceph-mon[35142]: pgmap v298: 129 pgs: 129 active+clean; 36 MiB data, 381 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-21T01:31:04.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 39060 4.33 MB/sec warmup 53 sec latency 12.831 ms 2024-09-21T01:31:04.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 127183 13.78 MB/sec warmup 51 sec latency 26.916 ms 2024-09-21T01:31:05.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 39638 4.28 MB/sec warmup 54 sec latency 12.154 ms 2024-09-21T01:31:05.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 130206 13.79 MB/sec warmup 52 sec latency 33.397 ms 2024-09-21T01:31:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:05 smithi045 ceph-mon[28361]: pgmap v299: 129 pgs: 129 active+clean; 36 MiB data, 382 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:31:06.195 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:05 smithi090 ceph-mon[35142]: pgmap v299: 129 pgs: 129 active+clean; 36 MiB data, 382 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:31:06.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 40344 4.25 MB/sec warmup 55 sec latency 16.048 ms 2024-09-21T01:31:06.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 132914 13.84 MB/sec warmup 53 sec latency 66.581 ms 2024-09-21T01:31:07.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 41201 4.23 MB/sec warmup 56 sec latency 14.933 ms 2024-09-21T01:31:07.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 135630 13.86 MB/sec warmup 54 sec latency 169.965 ms 2024-09-21T01:31:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:07 smithi045 ceph-mon[28361]: pgmap v300: 129 pgs: 129 active+clean; 38 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-09-21T01:31:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:07 smithi090 ceph-mon[35142]: pgmap v300: 129 pgs: 129 active+clean; 38 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-09-21T01:31:08.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:08.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 42050 4.28 MB/sec warmup 57 sec latency 13.713 ms 2024-09-21T01:31:08.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 138147 13.86 MB/sec warmup 55 sec latency 226.726 ms 2024-09-21T01:31:09.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 42738 4.30 MB/sec warmup 58 sec latency 15.261 ms 2024-09-21T01:31:09.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 141366 13.92 MB/sec warmup 56 sec latency 33.081 ms 2024-09-21T01:31:09.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:09 smithi090 ceph-mon[35142]: pgmap v301: 129 pgs: 129 active+clean; 37 MiB data, 394 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-21T01:31:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:09 smithi045 ceph-mon[28361]: pgmap v301: 129 pgs: 129 active+clean; 37 MiB data, 394 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-21T01:31:10.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 43263 4.25 MB/sec warmup 59 sec latency 17.701 ms 2024-09-21T01:31:10.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 144389 13.92 MB/sec warmup 57 sec latency 33.435 ms 2024-09-21T01:31:11.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 44069 4.26 MB/sec warmup 60 sec latency 12.585 ms 2024-09-21T01:31:11.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 147350 13.97 MB/sec warmup 58 sec latency 48.754 ms 2024-09-21T01:31:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:11 smithi045 ceph-mon[28361]: pgmap v302: 129 pgs: 129 active+clean; 37 MiB data, 401 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:31:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:11 smithi090 ceph-mon[35142]: pgmap v302: 129 pgs: 129 active+clean; 37 MiB data, 401 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:31:12.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 44906 4.22 MB/sec warmup 61 sec latency 12.325 ms 2024-09-21T01:31:12.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 149924 13.98 MB/sec warmup 59 sec latency 105.610 ms 2024-09-21T01:31:13.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 45876 4.27 MB/sec warmup 62 sec latency 13.954 ms 2024-09-21T01:31:13.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 153363 14.06 MB/sec warmup 60 sec latency 35.748 ms 2024-09-21T01:31:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:13 smithi045 ceph-mon[28361]: pgmap v303: 129 pgs: 129 active+clean; 39 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 102 op/s 2024-09-21T01:31:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:13 smithi090 ceph-mon[35142]: pgmap v303: 129 pgs: 129 active+clean; 39 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 102 op/s 2024-09-21T01:31:14.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 46380 4.27 MB/sec warmup 63 sec latency 82.582 ms 2024-09-21T01:31:14.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 154038 13.91 MB/sec warmup 61 sec latency 283.705 ms 2024-09-21T01:31:15.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 46848 4.22 MB/sec warmup 64 sec latency 56.685 ms 2024-09-21T01:31:15.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 154442 13.69 MB/sec warmup 62 sec latency 332.653 ms 2024-09-21T01:31:16.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:15 smithi045 ceph-mon[28361]: pgmap v304: 129 pgs: 129 active+clean; 39 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:31:16.195 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:15 smithi090 ceph-mon[35142]: pgmap v304: 129 pgs: 129 active+clean; 39 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:31:16.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 47539 4.22 MB/sec warmup 65 sec latency 69.201 ms 2024-09-21T01:31:16.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 154927 13.50 MB/sec warmup 63 sec latency 300.190 ms 2024-09-21T01:31:17.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 48290 4.17 MB/sec warmup 66 sec latency 37.298 ms 2024-09-21T01:31:17.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 155468 13.37 MB/sec warmup 64 sec latency 323.491 ms 2024-09-21T01:31:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:17 smithi045 ceph-mon[28361]: pgmap v305: 129 pgs: 129 active+clean; 39 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-21T01:31:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:17 smithi090 ceph-mon[35142]: pgmap v305: 129 pgs: 129 active+clean; 39 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-21T01:31:18.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 49039 4.20 MB/sec warmup 67 sec latency 96.117 ms 2024-09-21T01:31:18.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 155743 13.18 MB/sec warmup 65 sec latency 356.850 ms 2024-09-21T01:31:19.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 49681 4.22 MB/sec warmup 68 sec latency 80.948 ms 2024-09-21T01:31:19.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 155884 13.00 MB/sec warmup 66 sec latency 316.279 ms 2024-09-21T01:31:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:19 smithi045 ceph-mon[28361]: pgmap v306: 129 pgs: 129 active+clean; 40 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:31:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:19 smithi090 ceph-mon[35142]: pgmap v306: 129 pgs: 129 active+clean; 40 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:31:20.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 50214 4.18 MB/sec warmup 69 sec latency 40.624 ms 2024-09-21T01:31:20.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 156774 12.87 MB/sec warmup 67 sec latency 311.001 ms 2024-09-21T01:31:21.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 50943 4.15 MB/sec warmup 70 sec latency 16.233 ms 2024-09-21T01:31:21.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 158511 12.79 MB/sec warmup 68 sec latency 325.139 ms 2024-09-21T01:31:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:21 smithi045 ceph-mon[28361]: pgmap v307: 129 pgs: 129 active+clean; 41 MiB data, 408 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-21T01:31:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:21 smithi090 ceph-mon[35142]: pgmap v307: 129 pgs: 129 active+clean; 41 MiB data, 408 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-21T01:31:22.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 51771 4.15 MB/sec warmup 71 sec latency 18.144 ms 2024-09-21T01:31:22.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 161531 12.85 MB/sec warmup 69 sec latency 49.880 ms 2024-09-21T01:31:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:23.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 52625 4.18 MB/sec warmup 72 sec latency 13.541 ms 2024-09-21T01:31:23.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 164671 12.94 MB/sec warmup 70 sec latency 41.727 ms 2024-09-21T01:31:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:23 smithi045 ceph-mon[28361]: pgmap v308: 129 pgs: 129 active+clean; 41 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:31:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:23 smithi090 ceph-mon[35142]: pgmap v308: 129 pgs: 129 active+clean; 41 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:31:24.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 53366 4.20 MB/sec warmup 73 sec latency 10.863 ms 2024-09-21T01:31:24.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 167619 12.96 MB/sec warmup 71 sec latency 33.718 ms 2024-09-21T01:31:25.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 53903 4.16 MB/sec warmup 74 sec latency 16.521 ms 2024-09-21T01:31:25.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 171075 13.05 MB/sec warmup 72 sec latency 42.106 ms 2024-09-21T01:31:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:25 smithi045 ceph-mon[28361]: pgmap v309: 129 pgs: 129 active+clean; 40 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-09-21T01:31:26.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:25 smithi090 ceph-mon[35142]: pgmap v309: 129 pgs: 129 active+clean; 40 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 71 op/s 2024-09-21T01:31:26.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 54711 4.17 MB/sec warmup 75 sec latency 18.992 ms 2024-09-21T01:31:26.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 174560 13.13 MB/sec warmup 73 sec latency 43.530 ms 2024-09-21T01:31:27.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 55564 4.14 MB/sec warmup 76 sec latency 13.136 ms 2024-09-21T01:31:27.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 178052 13.21 MB/sec warmup 74 sec latency 33.673 ms 2024-09-21T01:31:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:27 smithi045 ceph-mon[28361]: pgmap v310: 129 pgs: 129 active+clean; 43 MiB data, 425 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:31:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:27 smithi090 ceph-mon[35142]: pgmap v310: 129 pgs: 129 active+clean; 43 MiB data, 425 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:31:28.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 56514 4.18 MB/sec warmup 77 sec latency 17.245 ms 2024-09-21T01:31:28.554 INFO:tasks.workunit.client.0.smithi045.stdout: 1 181589 13.29 MB/sec warmup 75 sec latency 32.932 ms 2024-09-21T01:31:29.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 57045 4.19 MB/sec warmup 78 sec latency 17.640 ms 2024-09-21T01:31:29.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 183911 13.27 MB/sec warmup 76 sec latency 91.741 ms 2024-09-21T01:31:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:29 smithi045 ceph-mon[28361]: pgmap v311: 129 pgs: 129 active+clean; 43 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:31:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:29 smithi090 ceph-mon[35142]: pgmap v311: 129 pgs: 129 active+clean; 43 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:31:30.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 57604 4.14 MB/sec warmup 79 sec latency 18.830 ms 2024-09-21T01:31:30.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 186833 13.28 MB/sec warmup 77 sec latency 45.602 ms 2024-09-21T01:31:31.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 58389 4.15 MB/sec warmup 80 sec latency 12.606 ms 2024-09-21T01:31:31.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 189896 13.33 MB/sec warmup 78 sec latency 33.451 ms 2024-09-21T01:31:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:31 smithi045 ceph-mon[28361]: pgmap v312: 129 pgs: 129 active+clean; 44 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 90 op/s 2024-09-21T01:31:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:31 smithi090 ceph-mon[35142]: pgmap v312: 129 pgs: 129 active+clean; 44 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 90 op/s 2024-09-21T01:31:32.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 59148 4.12 MB/sec warmup 81 sec latency 14.060 ms 2024-09-21T01:31:32.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 192470 13.35 MB/sec warmup 79 sec latency 83.832 ms 2024-09-21T01:31:33.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 60074 4.16 MB/sec warmup 82 sec latency 13.220 ms 2024-09-21T01:31:33.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 195904 13.42 MB/sec warmup 80 sec latency 42.096 ms 2024-09-21T01:31:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:33 smithi045 ceph-mon[28361]: pgmap v313: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:31:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:33 smithi090 ceph-mon[35142]: pgmap v313: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:31:34.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 60668 4.17 MB/sec warmup 83 sec latency 15.210 ms 2024-09-21T01:31:34.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 199373 13.48 MB/sec warmup 81 sec latency 100.162 ms 2024-09-21T01:31:35.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 61266 4.13 MB/sec warmup 84 sec latency 17.564 ms 2024-09-21T01:31:35.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 202864 13.55 MB/sec warmup 82 sec latency 46.572 ms 2024-09-21T01:31:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:35 smithi045 ceph-mon[28361]: pgmap v314: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:31:36.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:35 smithi090 ceph-mon[35142]: pgmap v314: 129 pgs: 129 active+clean; 44 MiB data, 438 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:31:36.496 INFO:tasks.workunit.client.1.smithi090.stdout: 1 62033 4.13 MB/sec warmup 85 sec latency 13.298 ms 2024-09-21T01:31:36.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 205489 13.56 MB/sec warmup 83 sec latency 33.696 ms 2024-09-21T01:31:37.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 62815 4.11 MB/sec warmup 86 sec latency 11.895 ms 2024-09-21T01:31:37.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 208601 13.59 MB/sec warmup 84 sec latency 70.277 ms 2024-09-21T01:31:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:37 smithi045 ceph-mon[28361]: pgmap v315: 129 pgs: 129 active+clean; 45 MiB data, 424 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-09-21T01:31:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:37 smithi090 ceph-mon[35142]: pgmap v315: 129 pgs: 129 active+clean; 45 MiB data, 424 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 114 op/s 2024-09-21T01:31:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:38.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 63652 4.14 MB/sec warmup 87 sec latency 17.210 ms 2024-09-21T01:31:38.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 210750 13.58 MB/sec warmup 85 sec latency 158.289 ms 2024-09-21T01:31:39.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 64281 4.15 MB/sec warmup 88 sec latency 14.514 ms 2024-09-21T01:31:39.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 213705 13.59 MB/sec warmup 86 sec latency 58.249 ms 2024-09-21T01:31:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:39 smithi045 ceph-mon[28361]: pgmap v316: 129 pgs: 129 active+clean; 45 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:31:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:39 smithi090 ceph-mon[35142]: pgmap v316: 129 pgs: 129 active+clean; 45 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:31:40.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 64924 4.12 MB/sec warmup 89 sec latency 15.839 ms 2024-09-21T01:31:40.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 217232 13.66 MB/sec warmup 87 sec latency 33.435 ms 2024-09-21T01:31:41.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 65696 4.12 MB/sec warmup 90 sec latency 33.572 ms 2024-09-21T01:31:41.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 220649 13.71 MB/sec warmup 88 sec latency 180.541 ms 2024-09-21T01:31:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:41 smithi045 ceph-mon[28361]: pgmap v317: 129 pgs: 129 active+clean; 45 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:31:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:41 smithi090 ceph-mon[35142]: pgmap v317: 129 pgs: 129 active+clean; 45 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:31:42.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 66368 4.10 MB/sec warmup 91 sec latency 89.030 ms 2024-09-21T01:31:42.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 222962 13.70 MB/sec warmup 89 sec latency 258.543 ms 2024-09-21T01:31:43.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 67199 4.13 MB/sec warmup 92 sec latency 12.720 ms 2024-09-21T01:31:43.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 225992 13.71 MB/sec warmup 90 sec latency 44.127 ms 2024-09-21T01:31:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:43 smithi045 ceph-mon[28361]: pgmap v318: 129 pgs: 129 active+clean; 45 MiB data, 426 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-21T01:31:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:43 smithi090 ceph-mon[35142]: pgmap v318: 129 pgs: 129 active+clean; 45 MiB data, 426 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-21T01:31:44.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 67844 4.13 MB/sec warmup 93 sec latency 12.491 ms 2024-09-21T01:31:44.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 228045 13.69 MB/sec warmup 91 sec latency 108.454 ms 2024-09-21T01:31:45.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 68513 4.11 MB/sec warmup 94 sec latency 19.320 ms 2024-09-21T01:31:45.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 231417 13.74 MB/sec warmup 92 sec latency 99.767 ms 2024-09-21T01:31:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:45 smithi045 ceph-mon[28361]: pgmap v319: 129 pgs: 129 active+clean; 45 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:31:46.196 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:45 smithi090 ceph-mon[35142]: pgmap v319: 129 pgs: 129 active+clean; 45 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:31:46.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 69335 4.10 MB/sec warmup 95 sec latency 11.439 ms 2024-09-21T01:31:46.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 234953 13.80 MB/sec warmup 93 sec latency 33.600 ms 2024-09-21T01:31:47.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 70136 4.12 MB/sec warmup 96 sec latency 15.452 ms 2024-09-21T01:31:47.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 236565 13.73 MB/sec warmup 94 sec latency 324.948 ms 2024-09-21T01:31:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:47 smithi045 ceph-mon[28361]: pgmap v320: 129 pgs: 129 active+clean; 48 MiB data, 421 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:31:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:47 smithi090 ceph-mon[35142]: pgmap v320: 129 pgs: 129 active+clean; 48 MiB data, 421 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:31:48.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 70807 4.11 MB/sec warmup 97 sec latency 53.779 ms 2024-09-21T01:31:48.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 237067 13.64 MB/sec warmup 95 sec latency 316.414 ms 2024-09-21T01:31:49.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 71383 4.12 MB/sec warmup 98 sec latency 80.136 ms 2024-09-21T01:31:49.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 237442 13.52 MB/sec warmup 96 sec latency 342.748 ms 2024-09-21T01:31:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:49 smithi045 ceph-mon[28361]: pgmap v321: 129 pgs: 129 active+clean; 49 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-09-21T01:31:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:49 smithi090 ceph-mon[35142]: pgmap v321: 129 pgs: 129 active+clean; 49 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 85 op/s 2024-09-21T01:31:50.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 71909 4.08 MB/sec warmup 99 sec latency 52.375 ms 2024-09-21T01:31:50.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 237484 13.38 MB/sec warmup 97 sec latency 325.125 ms 2024-09-21T01:31:51.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 72613 4.09 MB/sec warmup 100 sec latency 86.060 ms 2024-09-21T01:31:51.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 238380 13.29 MB/sec warmup 98 sec latency 291.722 ms 2024-09-21T01:31:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:51 smithi045 ceph-mon[28361]: pgmap v322: 129 pgs: 129 active+clean; 48 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-21T01:31:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:51 smithi090 ceph-mon[35142]: pgmap v322: 129 pgs: 129 active+clean; 48 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-21T01:31:52.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 73389 4.06 MB/sec warmup 101 sec latency 20.176 ms 2024-09-21T01:31:52.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 238431 13.16 MB/sec warmup 99 sec latency 291.802 ms 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:31:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:31:53.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:31:53.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:31:53.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:31:53.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:31:53.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:31:53.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:31:53.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:31:53.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 74285 4.10 MB/sec warmup 102 sec latency 81.573 ms 2024-09-21T01:31:53.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 238438 13.03 MB/sec warmup 100 sec latency 384.008 ms 2024-09-21T01:31:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:53 smithi045 ceph-mon[28361]: pgmap v323: 129 pgs: 129 active+clean; 48 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:31:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:53 smithi090 ceph-mon[35142]: pgmap v323: 129 pgs: 129 active+clean; 48 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:31:54.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 74837 4.10 MB/sec warmup 103 sec latency 36.045 ms 2024-09-21T01:31:54.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 239586 12.95 MB/sec warmup 101 sec latency 300.309 ms 2024-09-21T01:31:55.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:54 smithi045 ceph-mon[28361]: pgmap v324: 129 pgs: 129 active+clean; 49 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:31:55.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:54 smithi090 ceph-mon[35142]: pgmap v324: 129 pgs: 129 active+clean; 49 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:31:55.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 75416 4.07 MB/sec warmup 104 sec latency 15.942 ms 2024-09-21T01:31:55.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 242231 12.96 MB/sec warmup 102 sec latency 33.456 ms 2024-09-21T01:31:56.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 76190 4.07 MB/sec warmup 105 sec latency 13.246 ms 2024-09-21T01:31:56.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 245673 13.02 MB/sec warmup 103 sec latency 33.450 ms 2024-09-21T01:31:57.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:57 smithi045 ceph-mon[28361]: pgmap v325: 129 pgs: 129 active+clean; 49 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:31:57.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 76979 4.05 MB/sec warmup 106 sec latency 20.452 ms 2024-09-21T01:31:57.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:57 smithi090 ceph-mon[35142]: pgmap v325: 129 pgs: 129 active+clean; 49 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:31:57.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 249135 13.08 MB/sec warmup 104 sec latency 58.432 ms 2024-09-21T01:31:58.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 77854 4.08 MB/sec warmup 107 sec latency 13.453 ms 2024-09-21T01:31:58.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 252612 13.13 MB/sec warmup 105 sec latency 66.485 ms 2024-09-21T01:31:59.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:31:59 smithi045 ceph-mon[28361]: pgmap v326: 129 pgs: 129 active+clean; 51 MiB data, 438 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:31:59.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 78428 4.09 MB/sec warmup 108 sec latency 57.918 ms 2024-09-21T01:31:59.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:31:59 smithi090 ceph-mon[35142]: pgmap v326: 129 pgs: 129 active+clean; 51 MiB data, 438 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:31:59.555 INFO:tasks.workunit.client.0.smithi045.stdout: 1 254358 13.08 MB/sec warmup 106 sec latency 261.493 ms 2024-09-21T01:32:00.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 78954 4.06 MB/sec warmup 109 sec latency 54.153 ms 2024-09-21T01:32:00.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 254876 13.00 MB/sec warmup 107 sec latency 332.284 ms 2024-09-21T01:32:01.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:01 smithi045 ceph-mon[28361]: pgmap v327: 129 pgs: 129 active+clean; 50 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-09-21T01:32:01.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 79681 4.06 MB/sec warmup 110 sec latency 15.640 ms 2024-09-21T01:32:01.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:01 smithi090 ceph-mon[35142]: pgmap v327: 129 pgs: 129 active+clean; 50 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-09-21T01:32:01.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 257061 12.99 MB/sec warmup 108 sec latency 200.026 ms 2024-09-21T01:32:02.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 80449 4.04 MB/sec warmup 111 sec latency 15.170 ms 2024-09-21T01:32:02.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 259893 13.01 MB/sec warmup 109 sec latency 35.760 ms 2024-09-21T01:32:03.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:03 smithi045 ceph-mon[28361]: pgmap v328: 129 pgs: 129 active+clean; 50 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:32:03.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 81375 4.07 MB/sec warmup 112 sec latency 12.891 ms 2024-09-21T01:32:03.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:03 smithi090 ceph-mon[35142]: pgmap v328: 129 pgs: 129 active+clean; 50 MiB data, 440 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:32:03.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 263320 13.06 MB/sec warmup 110 sec latency 58.255 ms 2024-09-21T01:32:04.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 81922 4.08 MB/sec warmup 113 sec latency 12.809 ms 2024-09-21T01:32:04.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 266802 13.11 MB/sec warmup 111 sec latency 43.722 ms 2024-09-21T01:32:05.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:05 smithi045 ceph-mon[28361]: pgmap v329: 129 pgs: 129 active+clean; 52 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-21T01:32:05.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 82501 4.05 MB/sec warmup 114 sec latency 20.739 ms 2024-09-21T01:32:05.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:05 smithi090 ceph-mon[35142]: pgmap v329: 129 pgs: 129 active+clean; 52 MiB data, 465 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-21T01:32:05.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 270286 13.17 MB/sec warmup 112 sec latency 91.284 ms 2024-09-21T01:32:06.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 83230 4.05 MB/sec warmup 115 sec latency 46.407 ms 2024-09-21T01:32:06.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 270486 13.05 MB/sec warmup 113 sec latency 289.075 ms 2024-09-21T01:32:07.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:07 smithi045 ceph-mon[28361]: pgmap v330: 129 pgs: 129 active+clean; 51 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:32:07.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 83998 4.03 MB/sec warmup 116 sec latency 33.329 ms 2024-09-21T01:32:07.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:07 smithi090 ceph-mon[35142]: pgmap v330: 129 pgs: 129 active+clean; 51 MiB data, 484 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:32:07.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 271949 12.99 MB/sec warmup 114 sec latency 299.595 ms 2024-09-21T01:32:08.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:08.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 84922 4.06 MB/sec warmup 117 sec latency 13.919 ms 2024-09-21T01:32:08.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:08.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 274075 12.99 MB/sec warmup 115 sec latency 100.073 ms 2024-09-21T01:32:09.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:09 smithi045 ceph-mon[28361]: pgmap v331: 129 pgs: 129 active+clean; 53 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:32:09.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 85475 4.06 MB/sec warmup 118 sec latency 13.233 ms 2024-09-21T01:32:09.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:09 smithi090 ceph-mon[35142]: pgmap v331: 129 pgs: 129 active+clean; 53 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:32:09.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 277494 13.04 MB/sec warmup 116 sec latency 163.327 ms 2024-09-21T01:32:10.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 86050 4.04 MB/sec warmup 119 sec latency 17.756 ms 2024-09-21T01:32:10.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 280980 13.09 MB/sec warmup 117 sec latency 91.485 ms 2024-09-21T01:32:11.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:11 smithi045 ceph-mon[28361]: pgmap v332: 129 pgs: 129 active+clean; 53 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:32:11.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:11 smithi090 ceph-mon[35142]: pgmap v332: 129 pgs: 129 active+clean; 53 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:32:11.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 284470 13.14 MB/sec warmup 118 sec latency 41.472 ms 2024-09-21T01:32:12.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 87569 1.81 MB/sec execute 1 sec latency 14.537 ms 2024-09-21T01:32:12.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 287111 13.15 MB/sec warmup 119 sec latency 50.074 ms 2024-09-21T01:32:13.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:13 smithi045 ceph-mon[28361]: pgmap v333: 129 pgs: 129 active+clean; 53 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:32:13.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 88474 4.60 MB/sec execute 2 sec latency 38.810 ms 2024-09-21T01:32:13.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:13 smithi090 ceph-mon[35142]: pgmap v333: 129 pgs: 129 active+clean; 53 MiB data, 499 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:32:14.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 89004 4.54 MB/sec execute 3 sec latency 65.175 ms 2024-09-21T01:32:14.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 288219 0.50 MB/sec execute 1 sec latency 325.158 ms 2024-09-21T01:32:15.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:15 smithi045 ceph-mon[28361]: pgmap v334: 129 pgs: 129 active+clean; 55 MiB data, 506 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-21T01:32:15.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 89531 3.61 MB/sec execute 4 sec latency 60.754 ms 2024-09-21T01:32:15.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:15 smithi090 ceph-mon[35142]: pgmap v334: 129 pgs: 129 active+clean; 55 MiB data, 506 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-21T01:32:15.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 291613 9.63 MB/sec execute 2 sec latency 166.823 ms 2024-09-21T01:32:16.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 90321 3.82 MB/sec execute 5 sec latency 12.816 ms 2024-09-21T01:32:16.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 295107 12.75 MB/sec execute 3 sec latency 33.456 ms 2024-09-21T01:32:17.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:17 smithi045 ceph-mon[28361]: pgmap v335: 129 pgs: 129 active+clean; 56 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:32:17.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 91116 3.49 MB/sec execute 6 sec latency 14.867 ms 2024-09-21T01:32:17.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:17 smithi090 ceph-mon[35142]: pgmap v335: 129 pgs: 129 active+clean; 56 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:32:17.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 297734 13.11 MB/sec execute 4 sec latency 42.027 ms 2024-09-21T01:32:18.497 INFO:tasks.workunit.client.1.smithi090.stdout: 1 92025 4.05 MB/sec execute 7 sec latency 13.184 ms 2024-09-21T01:32:18.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 300736 13.63 MB/sec execute 5 sec latency 83.533 ms 2024-09-21T01:32:19.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:19 smithi045 ceph-mon[28361]: pgmap v336: 129 pgs: 129 active+clean; 56 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-21T01:32:19.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 92582 4.09 MB/sec execute 8 sec latency 82.789 ms 2024-09-21T01:32:19.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:19 smithi090 ceph-mon[35142]: pgmap v336: 129 pgs: 129 active+clean; 56 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-21T01:32:19.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 302335 12.80 MB/sec execute 6 sec latency 247.614 ms 2024-09-21T01:32:20.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 93103 3.73 MB/sec execute 9 sec latency 46.283 ms 2024-09-21T01:32:20.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 302449 11.01 MB/sec execute 7 sec latency 316.655 ms 2024-09-21T01:32:21.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:21 smithi045 ceph-mon[28361]: pgmap v337: 129 pgs: 129 active+clean; 56 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-21T01:32:21.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 93892 3.83 MB/sec execute 10 sec latency 12.660 ms 2024-09-21T01:32:21.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:21 smithi090 ceph-mon[35142]: pgmap v337: 129 pgs: 129 active+clean; 56 MiB data, 511 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-21T01:32:21.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 305807 11.97 MB/sec execute 8 sec latency 233.518 ms 2024-09-21T01:32:22.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 94679 3.65 MB/sec execute 11 sec latency 12.900 ms 2024-09-21T01:32:22.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 309347 12.76 MB/sec execute 9 sec latency 41.518 ms 2024-09-21T01:32:23.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:23 smithi045 ceph-mon[28361]: pgmap v338: 129 pgs: 129 active+clean; 56 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:32:23.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:23.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 95577 3.95 MB/sec execute 12 sec latency 15.877 ms 2024-09-21T01:32:23.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:23 smithi090 ceph-mon[35142]: pgmap v338: 129 pgs: 129 active+clean; 56 MiB data, 517 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:32:23.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:23.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 312832 13.38 MB/sec execute 10 sec latency 74.931 ms 2024-09-21T01:32:24.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 96178 4.01 MB/sec execute 13 sec latency 14.246 ms 2024-09-21T01:32:24.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 315482 13.46 MB/sec execute 11 sec latency 46.120 ms 2024-09-21T01:32:25.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:25 smithi045 ceph-mon[28361]: pgmap v339: 129 pgs: 129 active+clean; 57 MiB data, 522 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:32:25.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 96783 3.79 MB/sec execute 14 sec latency 16.454 ms 2024-09-21T01:32:25.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:25 smithi090 ceph-mon[35142]: pgmap v339: 129 pgs: 129 active+clean; 57 MiB data, 522 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:32:25.556 INFO:tasks.workunit.client.0.smithi045.stdout: 1 318082 13.31 MB/sec execute 12 sec latency 158.548 ms 2024-09-21T01:32:26.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 97466 3.83 MB/sec execute 15 sec latency 79.278 ms 2024-09-21T01:32:26.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 319928 13.24 MB/sec execute 13 sec latency 308.478 ms 2024-09-21T01:32:27.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:27 smithi045 ceph-mon[28361]: pgmap v340: 129 pgs: 129 active+clean; 58 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:32:27.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 98219 3.70 MB/sec execute 16 sec latency 43.742 ms 2024-09-21T01:32:27.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:27 smithi090 ceph-mon[35142]: pgmap v340: 129 pgs: 129 active+clean; 58 MiB data, 524 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:32:27.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 320054 12.31 MB/sec execute 14 sec latency 308.501 ms 2024-09-21T01:32:28.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 99123 3.92 MB/sec execute 17 sec latency 13.185 ms 2024-09-21T01:32:28.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 323597 12.77 MB/sec execute 15 sec latency 32.969 ms 2024-09-21T01:32:29.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:29 smithi045 ceph-mon[28361]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:32:29.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 99737 3.96 MB/sec execute 18 sec latency 12.991 ms 2024-09-21T01:32:29.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:29 smithi090 ceph-mon[35142]: pgmap v341: 129 pgs: 129 active+clean; 59 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:32:29.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 327077 13.15 MB/sec execute 16 sec latency 33.007 ms 2024-09-21T01:32:30.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 100323 3.80 MB/sec execute 19 sec latency 25.196 ms 2024-09-21T01:32:30.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 330571 13.50 MB/sec execute 17 sec latency 108.468 ms 2024-09-21T01:32:31.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 101152 3.83 MB/sec execute 20 sec latency 14.330 ms 2024-09-21T01:32:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:31 smithi090 ceph-mon[35142]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:32:31.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 334105 13.81 MB/sec execute 18 sec latency 110.697 ms 2024-09-21T01:32:31.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:31 smithi045 ceph-mon[28361]: pgmap v342: 129 pgs: 129 active+clean; 59 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:32:32.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 101920 3.77 MB/sec execute 21 sec latency 20.792 ms 2024-09-21T01:32:32.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 336730 13.83 MB/sec execute 19 sec latency 33.284 ms 2024-09-21T01:32:33.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 102709 3.90 MB/sec execute 22 sec latency 13.022 ms 2024-09-21T01:32:33.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:33 smithi090 ceph-mon[35142]: pgmap v343: 129 pgs: 129 active+clean; 59 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:32:33.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 339956 14.00 MB/sec execute 20 sec latency 32.306 ms 2024-09-21T01:32:33.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:33 smithi045 ceph-mon[28361]: pgmap v343: 129 pgs: 129 active+clean; 59 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:32:34.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 103387 3.93 MB/sec execute 23 sec latency 9.867 ms 2024-09-21T01:32:34.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 342997 14.02 MB/sec execute 21 sec latency 27.232 ms 2024-09-21T01:32:35.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:35 smithi090 ceph-mon[35142]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:32:35.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 104086 3.85 MB/sec execute 24 sec latency 15.384 ms 2024-09-21T01:32:35.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 345957 14.14 MB/sec execute 22 sec latency 63.773 ms 2024-09-21T01:32:35.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:35 smithi045 ceph-mon[28361]: pgmap v344: 129 pgs: 129 active+clean; 59 MiB data, 539 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:32:36.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 104903 3.84 MB/sec execute 25 sec latency 18.255 ms 2024-09-21T01:32:36.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 348600 14.16 MB/sec execute 23 sec latency 58.451 ms 2024-09-21T01:32:37.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:37 smithi090 ceph-mon[35142]: pgmap v345: 129 pgs: 129 active+clean; 61 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-21T01:32:37.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 105779 3.91 MB/sec execute 26 sec latency 10.720 ms 2024-09-21T01:32:37.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 351970 14.35 MB/sec execute 24 sec latency 43.994 ms 2024-09-21T01:32:37.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:37 smithi045 ceph-mon[28361]: pgmap v345: 129 pgs: 129 active+clean; 61 MiB data, 546 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-21T01:32:38.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:38.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 106410 3.99 MB/sec execute 27 sec latency 14.609 ms 2024-09-21T01:32:38.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 355440 14.54 MB/sec execute 25 sec latency 100.080 ms 2024-09-21T01:32:38.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:39.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:39 smithi090 ceph-mon[35142]: pgmap v346: 129 pgs: 129 active+clean; 61 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:32:39.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 106963 3.91 MB/sec execute 28 sec latency 12.653 ms 2024-09-21T01:32:39.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 358037 14.52 MB/sec execute 26 sec latency 108.057 ms 2024-09-21T01:32:39.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:39 smithi045 ceph-mon[28361]: pgmap v346: 129 pgs: 129 active+clean; 61 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:32:40.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 107637 3.85 MB/sec execute 29 sec latency 20.414 ms 2024-09-21T01:32:40.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 361246 14.62 MB/sec execute 27 sec latency 50.028 ms 2024-09-21T01:32:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:41 smithi090 ceph-mon[35142]: pgmap v347: 129 pgs: 129 active+clean; 60 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-09-21T01:32:41.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 108412 3.83 MB/sec execute 30 sec latency 14.427 ms 2024-09-21T01:32:41.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 364169 14.58 MB/sec execute 28 sec latency 100.111 ms 2024-09-21T01:32:41.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:41 smithi045 ceph-mon[28361]: pgmap v347: 129 pgs: 129 active+clean; 60 MiB data, 562 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-09-21T01:32:42.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 109209 3.89 MB/sec execute 31 sec latency 12.535 ms 2024-09-21T01:32:42.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 367230 14.68 MB/sec execute 29 sec latency 41.505 ms 2024-09-21T01:32:43.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:43 smithi090 ceph-mon[35142]: pgmap v348: 129 pgs: 129 active+clean; 62 MiB data, 548 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-21T01:32:43.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 110004 3.97 MB/sec execute 32 sec latency 12.203 ms 2024-09-21T01:32:43.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 369876 14.68 MB/sec execute 30 sec latency 33.672 ms 2024-09-21T01:32:43.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:43 smithi045 ceph-mon[28361]: pgmap v348: 129 pgs: 129 active+clean; 62 MiB data, 548 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-21T01:32:44.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 110558 3.90 MB/sec execute 33 sec latency 9.979 ms 2024-09-21T01:32:44.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 373315 14.82 MB/sec execute 31 sec latency 53.963 ms 2024-09-21T01:32:45.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 111256 3.85 MB/sec execute 34 sec latency 17.703 ms 2024-09-21T01:32:45.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 376780 14.95 MB/sec execute 32 sec latency 33.683 ms 2024-09-21T01:32:45.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:45 smithi045 ceph-mon[28361]: pgmap v349: 129 pgs: 129 active+clean; 63 MiB data, 548 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:32:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:45 smithi090 ceph-mon[35142]: pgmap v349: 129 pgs: 129 active+clean; 63 MiB data, 548 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:32:46.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 112064 3.84 MB/sec execute 35 sec latency 14.334 ms 2024-09-21T01:32:46.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 380304 15.07 MB/sec execute 33 sec latency 41.700 ms 2024-09-21T01:32:47.498 INFO:tasks.workunit.client.1.smithi090.stdout: 1 112934 3.92 MB/sec execute 36 sec latency 16.200 ms 2024-09-21T01:32:47.557 INFO:tasks.workunit.client.0.smithi045.stdout: 1 383837 15.19 MB/sec execute 34 sec latency 26.000 ms 2024-09-21T01:32:47.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:47 smithi045 ceph-mon[28361]: pgmap v350: 129 pgs: 129 active+clean; 65 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:32:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:47 smithi090 ceph-mon[35142]: pgmap v350: 129 pgs: 129 active+clean; 65 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:32:48.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 113671 3.96 MB/sec execute 37 sec latency 12.506 ms 2024-09-21T01:32:48.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 385895 15.06 MB/sec execute 35 sec latency 241.647 ms 2024-09-21T01:32:49.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 114201 3.89 MB/sec execute 38 sec latency 17.501 ms 2024-09-21T01:32:49.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 388314 15.01 MB/sec execute 36 sec latency 116.733 ms 2024-09-21T01:32:49.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:49 smithi045 ceph-mon[28361]: pgmap v351: 129 pgs: 129 active+clean; 64 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:32:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:49 smithi090 ceph-mon[35142]: pgmap v351: 129 pgs: 129 active+clean; 64 MiB data, 551 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:32:50.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 114978 3.92 MB/sec execute 39 sec latency 17.487 ms 2024-09-21T01:32:50.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 391209 15.01 MB/sec execute 37 sec latency 27.087 ms 2024-09-21T01:32:51.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 115767 3.84 MB/sec execute 40 sec latency 14.011 ms 2024-09-21T01:32:51.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 394576 15.11 MB/sec execute 38 sec latency 74.484 ms 2024-09-21T01:32:51.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:51 smithi045 ceph-mon[28361]: pgmap v352: 129 pgs: 129 active+clean; 65 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:32:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:51 smithi090 ceph-mon[35142]: pgmap v352: 129 pgs: 129 active+clean; 65 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:32:52.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 116707 3.93 MB/sec execute 41 sec latency 13.619 ms 2024-09-21T01:32:52.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 398110 15.21 MB/sec execute 39 sec latency 33.405 ms 2024-09-21T01:32:52.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:32:53.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 117312 3.97 MB/sec execute 42 sec latency 13.463 ms 2024-09-21T01:32:53.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 401547 15.30 MB/sec execute 40 sec latency 51.960 ms 2024-09-21T01:32:53.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:53 smithi045 ceph-mon[28361]: pgmap v353: 129 pgs: 129 active+clean; 65 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:32:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:32:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:53 smithi090 ceph-mon[35142]: pgmap v353: 129 pgs: 129 active+clean; 65 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:32:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:32:54.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 117778 3.89 MB/sec execute 43 sec latency 17.392 ms 2024-09-21T01:32:54.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 401598 14.93 MB/sec execute 41 sec latency 283.250 ms 2024-09-21T01:32:54.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:32:54.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:32:54.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:32:54.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:32:54.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:32:54.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:32:55.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 118494 3.91 MB/sec execute 44 sec latency 81.920 ms 2024-09-21T01:32:55.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 401659 14.58 MB/sec execute 42 sec latency 291.840 ms 2024-09-21T01:32:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:55 smithi090 ceph-mon[35142]: pgmap v354: 129 pgs: 129 active+clean; 66 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:32:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:55 smithi045 ceph-mon[28361]: pgmap v354: 129 pgs: 129 active+clean; 66 MiB data, 596 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:32:56.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 119253 3.84 MB/sec execute 45 sec latency 46.659 ms 2024-09-21T01:32:56.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 401668 14.24 MB/sec execute 43 sec latency 333.469 ms 2024-09-21T01:32:57.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 120079 3.90 MB/sec execute 46 sec latency 41.833 ms 2024-09-21T01:32:57.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 401745 13.92 MB/sec execute 44 sec latency 308.451 ms 2024-09-21T01:32:57.971 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:57 smithi045 ceph-mon[28361]: pgmap v355: 129 pgs: 129 active+clean; 66 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-21T01:32:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:57 smithi090 ceph-mon[35142]: pgmap v355: 129 pgs: 129 active+clean; 66 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-21T01:32:58.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 120744 3.93 MB/sec execute 47 sec latency 62.988 ms 2024-09-21T01:32:58.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 401848 13.61 MB/sec execute 45 sec latency 345.311 ms 2024-09-21T01:32:59.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 121204 3.88 MB/sec execute 48 sec latency 40.127 ms 2024-09-21T01:32:59.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 402499 13.41 MB/sec execute 46 sec latency 297.486 ms 2024-09-21T01:32:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:32:59 smithi090 ceph-mon[35142]: pgmap v356: 129 pgs: 129 active+clean; 67 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:33:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:32:59 smithi045 ceph-mon[28361]: pgmap v356: 129 pgs: 129 active+clean; 67 MiB data, 579 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:33:00.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 121742 3.84 MB/sec execute 49 sec latency 97.906 ms 2024-09-21T01:33:00.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 402753 13.13 MB/sec execute 47 sec latency 325.181 ms 2024-09-21T01:33:01.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 122506 3.83 MB/sec execute 50 sec latency 100.313 ms 2024-09-21T01:33:01.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 405201 13.15 MB/sec execute 48 sec latency 291.898 ms 2024-09-21T01:33:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:01 smithi090 ceph-mon[35142]: pgmap v357: 129 pgs: 129 active+clean; 66 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-09-21T01:33:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:01 smithi045 ceph-mon[28361]: pgmap v357: 129 pgs: 129 active+clean; 66 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 84 op/s 2024-09-21T01:33:02.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 123193 3.80 MB/sec execute 51 sec latency 68.330 ms 2024-09-21T01:33:02.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 408683 13.27 MB/sec execute 49 sec latency 41.777 ms 2024-09-21T01:33:03.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 123964 3.85 MB/sec execute 52 sec latency 13.025 ms 2024-09-21T01:33:03.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 412175 13.39 MB/sec execute 50 sec latency 48.488 ms 2024-09-21T01:33:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:03 smithi090 ceph-mon[35142]: pgmap v358: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:33:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:03 smithi045 ceph-mon[28361]: pgmap v358: 129 pgs: 129 active+clean; 67 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:33:04.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 124605 3.87 MB/sec execute 53 sec latency 29.997 ms 2024-09-21T01:33:04.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 414412 13.36 MB/sec execute 51 sec latency 183.537 ms 2024-09-21T01:33:05.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 125251 3.83 MB/sec execute 54 sec latency 16.740 ms 2024-09-21T01:33:05.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 417502 13.38 MB/sec execute 52 sec latency 33.661 ms 2024-09-21T01:33:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:05 smithi090 ceph-mon[35142]: pgmap v359: 129 pgs: 129 active+clean; 68 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-21T01:33:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:05 smithi045 ceph-mon[28361]: pgmap v359: 129 pgs: 129 active+clean; 68 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-21T01:33:06.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 126049 3.83 MB/sec execute 55 sec latency 13.948 ms 2024-09-21T01:33:06.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 419470 13.36 MB/sec execute 53 sec latency 162.181 ms 2024-09-21T01:33:07.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 126766 3.81 MB/sec execute 56 sec latency 35.125 ms 2024-09-21T01:33:07.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 419576 13.12 MB/sec execute 54 sec latency 299.962 ms 2024-09-21T01:33:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:07 smithi090 ceph-mon[35142]: pgmap v360: 129 pgs: 129 active+clean; 67 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:33:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:07 smithi045 ceph-mon[28361]: pgmap v360: 129 pgs: 129 active+clean; 67 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:33:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:08.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 127513 3.85 MB/sec execute 57 sec latency 82.062 ms 2024-09-21T01:33:08.558 INFO:tasks.workunit.client.0.smithi045.stdout: 1 422876 13.21 MB/sec execute 55 sec latency 275.086 ms 2024-09-21T01:33:09.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 128150 3.87 MB/sec execute 58 sec latency 13.296 ms 2024-09-21T01:33:09.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 426357 13.32 MB/sec execute 56 sec latency 74.450 ms 2024-09-21T01:33:09.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:09 smithi090 ceph-mon[35142]: pgmap v361: 129 pgs: 129 active+clean; 68 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:33:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:09 smithi045 ceph-mon[28361]: pgmap v361: 129 pgs: 129 active+clean; 68 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:33:10.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 128786 3.83 MB/sec execute 59 sec latency 16.176 ms 2024-09-21T01:33:10.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 428869 13.31 MB/sec execute 57 sec latency 107.814 ms 2024-09-21T01:33:11.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 129574 3.83 MB/sec execute 60 sec latency 15.132 ms 2024-09-21T01:33:11.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 431700 13.32 MB/sec execute 58 sec latency 66.806 ms 2024-09-21T01:33:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:11 smithi090 ceph-mon[35142]: pgmap v362: 129 pgs: 129 active+clean; 68 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-21T01:33:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:11 smithi045 ceph-mon[28361]: pgmap v362: 129 pgs: 129 active+clean; 68 MiB data, 590 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-21T01:33:12.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 130276 3.81 MB/sec execute 61 sec latency 21.337 ms 2024-09-21T01:33:12.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 434406 13.37 MB/sec execute 59 sec latency 105.626 ms 2024-09-21T01:33:13.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 131052 3.85 MB/sec execute 62 sec latency 46.275 ms 2024-09-21T01:33:13.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 436996 13.39 MB/sec execute 60 sec latency 283.536 ms 2024-09-21T01:33:13.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:13 smithi090 ceph-mon[35142]: pgmap v363: 129 pgs: 129 active+clean; 69 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:33:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:13 smithi045 ceph-mon[28361]: pgmap v363: 129 pgs: 129 active+clean; 69 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:33:14.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 131652 3.86 MB/sec execute 63 sec latency 50.300 ms 2024-09-21T01:33:14.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 437057 13.17 MB/sec execute 61 sec latency 316.756 ms 2024-09-21T01:33:15.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 132256 3.82 MB/sec execute 64 sec latency 16.308 ms 2024-09-21T01:33:15.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 440588 13.26 MB/sec execute 62 sec latency 41.762 ms 2024-09-21T01:33:15.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:15 smithi045 ceph-mon[28361]: pgmap v364: 129 pgs: 129 active+clean; 70 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:33:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:15 smithi090 ceph-mon[35142]: pgmap v364: 129 pgs: 129 active+clean; 70 MiB data, 601 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:33:16.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 133018 3.83 MB/sec execute 65 sec latency 15.938 ms 2024-09-21T01:33:16.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 444083 13.35 MB/sec execute 63 sec latency 33.436 ms 2024-09-21T01:33:17.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 133775 3.81 MB/sec execute 66 sec latency 11.293 ms 2024-09-21T01:33:17.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 446698 13.37 MB/sec execute 64 sec latency 55.144 ms 2024-09-21T01:33:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:17 smithi090 ceph-mon[35142]: pgmap v365: 129 pgs: 129 active+clean; 71 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:33:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:17 smithi045 ceph-mon[28361]: pgmap v365: 129 pgs: 129 active+clean; 71 MiB data, 607 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:33:18.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 134598 3.85 MB/sec execute 67 sec latency 13.085 ms 2024-09-21T01:33:18.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 449866 13.43 MB/sec execute 65 sec latency 33.623 ms 2024-09-21T01:33:19.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 135236 3.86 MB/sec execute 68 sec latency 14.063 ms 2024-09-21T01:33:19.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 452872 13.43 MB/sec execute 66 sec latency 33.639 ms 2024-09-21T01:33:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:19 smithi090 ceph-mon[35142]: pgmap v366: 129 pgs: 129 active+clean; 72 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:33:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:19 smithi045 ceph-mon[28361]: pgmap v366: 129 pgs: 129 active+clean; 72 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:33:20.499 INFO:tasks.workunit.client.1.smithi090.stdout: 1 135880 3.83 MB/sec execute 69 sec latency 15.861 ms 2024-09-21T01:33:20.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 455928 13.49 MB/sec execute 67 sec latency 75.127 ms 2024-09-21T01:33:21.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 136695 3.83 MB/sec execute 70 sec latency 15.106 ms 2024-09-21T01:33:21.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 459140 13.81 MB/sec execute 68 sec latency 33.497 ms 2024-09-21T01:33:21.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:21 smithi090 ceph-mon[35142]: pgmap v367: 129 pgs: 129 active+clean; 71 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-09-21T01:33:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:21 smithi045 ceph-mon[28361]: pgmap v367: 129 pgs: 129 active+clean; 71 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-09-21T01:33:22.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 137438 3.81 MB/sec execute 71 sec latency 19.729 ms 2024-09-21T01:33:22.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 462645 13.89 MB/sec execute 69 sec latency 41.347 ms 2024-09-21T01:33:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:23.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 138311 3.89 MB/sec execute 72 sec latency 13.212 ms 2024-09-21T01:33:23.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 466134 13.96 MB/sec execute 70 sec latency 35.095 ms 2024-09-21T01:33:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:23 smithi045 ceph-mon[28361]: pgmap v368: 129 pgs: 129 active+clean; 72 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:33:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:23 smithi090 ceph-mon[35142]: pgmap v368: 129 pgs: 129 active+clean; 72 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:33:24.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 138871 3.86 MB/sec execute 73 sec latency 14.687 ms 2024-09-21T01:33:24.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 469671 14.03 MB/sec execute 71 sec latency 33.467 ms 2024-09-21T01:33:25.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 139563 3.83 MB/sec execute 74 sec latency 17.540 ms 2024-09-21T01:33:25.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 471994 14.01 MB/sec execute 72 sec latency 66.892 ms 2024-09-21T01:33:26.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:25 smithi045 ceph-mon[28361]: pgmap v369: 129 pgs: 129 active+clean; 72 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:33:26.199 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:25 smithi090 ceph-mon[35142]: pgmap v369: 129 pgs: 129 active+clean; 72 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:33:26.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 140402 3.83 MB/sec execute 75 sec latency 17.000 ms 2024-09-21T01:33:26.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 474916 14.00 MB/sec execute 73 sec latency 39.898 ms 2024-09-21T01:33:27.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 141143 3.85 MB/sec execute 76 sec latency 17.511 ms 2024-09-21T01:33:27.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 477576 14.05 MB/sec execute 74 sec latency 33.433 ms 2024-09-21T01:33:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:27 smithi045 ceph-mon[28361]: pgmap v370: 129 pgs: 129 active+clean; 74 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:33:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:27 smithi090 ceph-mon[35142]: pgmap v370: 129 pgs: 129 active+clean; 74 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:33:28.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 141921 3.88 MB/sec execute 77 sec latency 14.023 ms 2024-09-21T01:33:28.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 480481 14.06 MB/sec execute 75 sec latency 82.079 ms 2024-09-21T01:33:29.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 142457 3.86 MB/sec execute 78 sec latency 12.502 ms 2024-09-21T01:33:29.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 484021 14.12 MB/sec execute 76 sec latency 33.568 ms 2024-09-21T01:33:30.050 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:29 smithi090 ceph-mon[35142]: pgmap v371: 129 pgs: 129 active+clean; 74 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:33:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:29 smithi045 ceph-mon[28361]: pgmap v371: 129 pgs: 129 active+clean; 74 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:33:30.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 143099 3.83 MB/sec execute 79 sec latency 21.136 ms 2024-09-21T01:33:30.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 487500 14.18 MB/sec execute 77 sec latency 33.684 ms 2024-09-21T01:33:31.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 143919 3.83 MB/sec execute 80 sec latency 13.677 ms 2024-09-21T01:33:31.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 490992 14.25 MB/sec execute 78 sec latency 61.306 ms 2024-09-21T01:33:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:31 smithi045 ceph-mon[28361]: pgmap v372: 129 pgs: 129 active+clean; 75 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-21T01:33:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:31 smithi090 ceph-mon[35142]: pgmap v372: 129 pgs: 129 active+clean; 75 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-21T01:33:32.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 144803 3.86 MB/sec execute 81 sec latency 12.126 ms 2024-09-21T01:33:32.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 493649 14.25 MB/sec execute 79 sec latency 33.433 ms 2024-09-21T01:33:33.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 145457 3.88 MB/sec execute 82 sec latency 14.177 ms 2024-09-21T01:33:33.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 497153 14.31 MB/sec execute 80 sec latency 33.625 ms 2024-09-21T01:33:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:33 smithi045 ceph-mon[28361]: pgmap v373: 129 pgs: 129 active+clean; 75 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:33:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:33 smithi090 ceph-mon[35142]: pgmap v373: 129 pgs: 129 active+clean; 75 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:33:34.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 146025 3.85 MB/sec execute 83 sec latency 12.154 ms 2024-09-21T01:33:34.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 500167 14.32 MB/sec execute 81 sec latency 87.564 ms 2024-09-21T01:33:35.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 146721 3.83 MB/sec execute 84 sec latency 15.356 ms 2024-09-21T01:33:35.559 INFO:tasks.workunit.client.0.smithi045.stdout: 1 501751 14.25 MB/sec execute 82 sec latency 266.577 ms 2024-09-21T01:33:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:35 smithi045 ceph-mon[28361]: pgmap v374: 129 pgs: 129 active+clean; 74 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:33:36.199 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:35 smithi090 ceph-mon[35142]: pgmap v374: 129 pgs: 129 active+clean; 74 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:33:36.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 147560 3.83 MB/sec execute 85 sec latency 12.185 ms 2024-09-21T01:33:36.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 505294 14.31 MB/sec execute 83 sec latency 41.516 ms 2024-09-21T01:33:37.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 148435 3.87 MB/sec execute 86 sec latency 12.718 ms 2024-09-21T01:33:37.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 508718 14.37 MB/sec execute 84 sec latency 99.722 ms 2024-09-21T01:33:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:37 smithi045 ceph-mon[28361]: pgmap v375: 129 pgs: 129 active+clean; 77 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:33:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:37 smithi090 ceph-mon[35142]: pgmap v375: 129 pgs: 129 active+clean; 77 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:33:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:38.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 149161 3.88 MB/sec execute 87 sec latency 14.081 ms 2024-09-21T01:33:38.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 512259 14.42 MB/sec execute 85 sec latency 26.736 ms 2024-09-21T01:33:39.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 149695 3.85 MB/sec execute 88 sec latency 15.891 ms 2024-09-21T01:33:39.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 514907 14.42 MB/sec execute 86 sec latency 33.079 ms 2024-09-21T01:33:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:39 smithi045 ceph-mon[28361]: pgmap v376: 129 pgs: 129 active+clean; 77 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:33:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:39 smithi090 ceph-mon[35142]: pgmap v376: 129 pgs: 129 active+clean; 77 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:33:40.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 150495 3.87 MB/sec execute 89 sec latency 12.632 ms 2024-09-21T01:33:40.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 518102 14.45 MB/sec execute 87 sec latency 80.955 ms 2024-09-21T01:33:41.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 151212 3.83 MB/sec execute 90 sec latency 48.415 ms 2024-09-21T01:33:41.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 518449 14.31 MB/sec execute 88 sec latency 308.374 ms 2024-09-21T01:33:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:41 smithi045 ceph-mon[28361]: pgmap v377: 129 pgs: 129 active+clean; 78 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-09-21T01:33:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:41 smithi090 ceph-mon[35142]: pgmap v377: 129 pgs: 129 active+clean; 78 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-09-21T01:33:42.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 152017 3.86 MB/sec execute 91 sec latency 58.956 ms 2024-09-21T01:33:42.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 519351 14.20 MB/sec execute 89 sec latency 288.032 ms 2024-09-21T01:33:43.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 152751 3.89 MB/sec execute 92 sec latency 20.242 ms 2024-09-21T01:33:43.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 519361 14.05 MB/sec execute 90 sec latency 275.130 ms 2024-09-21T01:33:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:43 smithi045 ceph-mon[28361]: pgmap v378: 129 pgs: 129 active+clean; 78 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:33:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:43 smithi090 ceph-mon[35142]: pgmap v378: 129 pgs: 129 active+clean; 78 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:33:44.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 153167 3.85 MB/sec execute 93 sec latency 60.828 ms 2024-09-21T01:33:44.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 519412 13.89 MB/sec execute 91 sec latency 316.818 ms 2024-09-21T01:33:45.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 153814 3.83 MB/sec execute 94 sec latency 38.290 ms 2024-09-21T01:33:45.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 519473 13.74 MB/sec execute 92 sec latency 300.192 ms 2024-09-21T01:33:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:45 smithi045 ceph-mon[28361]: pgmap v379: 129 pgs: 129 active+clean; 78 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:33:46.200 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:45 smithi090 ceph-mon[35142]: pgmap v379: 129 pgs: 129 active+clean; 78 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:33:46.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 154606 3.83 MB/sec execute 95 sec latency 56.603 ms 2024-09-21T01:33:46.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 519482 13.60 MB/sec execute 93 sec latency 316.827 ms 2024-09-21T01:33:47.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 155442 3.85 MB/sec execute 96 sec latency 18.121 ms 2024-09-21T01:33:47.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 519596 13.45 MB/sec execute 94 sec latency 275.249 ms 2024-09-21T01:33:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:47 smithi045 ceph-mon[28361]: pgmap v380: 129 pgs: 129 active+clean; 79 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:33:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:47 smithi090 ceph-mon[35142]: pgmap v380: 129 pgs: 129 active+clean; 79 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:33:48.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 156116 3.87 MB/sec execute 97 sec latency 26.218 ms 2024-09-21T01:33:48.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 522899 13.51 MB/sec execute 95 sec latency 283.222 ms 2024-09-21T01:33:49.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 156685 3.85 MB/sec execute 98 sec latency 10.900 ms 2024-09-21T01:33:49.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 526443 13.57 MB/sec execute 96 sec latency 45.265 ms 2024-09-21T01:33:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:49 smithi045 ceph-mon[28361]: pgmap v381: 129 pgs: 129 active+clean; 78 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:33:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:49 smithi090 ceph-mon[35142]: pgmap v381: 129 pgs: 129 active+clean; 78 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:33:50.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 157481 3.86 MB/sec execute 99 sec latency 22.902 ms 2024-09-21T01:33:50.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 528761 13.55 MB/sec execute 97 sec latency 91.476 ms 2024-09-21T01:33:51.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 158257 3.83 MB/sec execute 100 sec latency 16.362 ms 2024-09-21T01:33:51.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 530947 13.54 MB/sec execute 98 sec latency 199.154 ms 2024-09-21T01:33:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:51 smithi045 ceph-mon[28361]: pgmap v382: 129 pgs: 129 active+clean; 79 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 95 op/s 2024-09-21T01:33:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:51 smithi090 ceph-mon[35142]: pgmap v382: 129 pgs: 129 active+clean; 79 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 95 op/s 2024-09-21T01:33:52.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 159086 3.86 MB/sec execute 101 sec latency 13.008 ms 2024-09-21T01:33:52.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 533700 13.55 MB/sec execute 99 sec latency 264.297 ms 2024-09-21T01:33:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:33:53.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 159754 3.87 MB/sec execute 102 sec latency 55.779 ms 2024-09-21T01:33:53.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 535808 13.53 MB/sec execute 100 sec latency 331.409 ms 2024-09-21T01:33:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:53 smithi045 ceph-mon[28361]: pgmap v383: 129 pgs: 129 active+clean; 81 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:33:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:53 smithi090 ceph-mon[35142]: pgmap v383: 129 pgs: 129 active+clean; 81 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-21T01:33:54.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 160239 3.85 MB/sec execute 103 sec latency 49.576 ms 2024-09-21T01:33:54.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 537114 13.46 MB/sec execute 101 sec latency 349.853 ms 2024-09-21T01:33:55.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:33:55.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:33:55.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:54 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:33:55.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:33:55.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:33:55.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:54 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:33:55.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 160898 3.83 MB/sec execute 104 sec latency 25.412 ms 2024-09-21T01:33:55.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 539732 13.47 MB/sec execute 102 sec latency 74.404 ms 2024-09-21T01:33:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:55 smithi045 ceph-mon[28361]: pgmap v384: 129 pgs: 129 active+clean; 81 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:33:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:33:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:33:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:33:56.200 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:55 smithi090 ceph-mon[35142]: pgmap v384: 129 pgs: 129 active+clean; 81 MiB data, 681 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:33:56.200 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:33:56.200 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:33:56.200 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:33:56.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 161735 3.83 MB/sec execute 105 sec latency 13.027 ms 2024-09-21T01:33:56.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 542898 13.51 MB/sec execute 103 sec latency 33.424 ms 2024-09-21T01:33:57.500 INFO:tasks.workunit.client.1.smithi090.stdout: 1 162617 3.86 MB/sec execute 106 sec latency 17.728 ms 2024-09-21T01:33:57.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 545412 13.49 MB/sec execute 104 sec latency 60.099 ms 2024-09-21T01:33:58.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:57 smithi045 ceph-mon[28361]: pgmap v385: 129 pgs: 129 active+clean; 82 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 106 op/s 2024-09-21T01:33:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:57 smithi090 ceph-mon[35142]: pgmap v385: 129 pgs: 129 active+clean; 82 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 106 op/s 2024-09-21T01:33:58.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 163336 3.87 MB/sec execute 107 sec latency 14.088 ms 2024-09-21T01:33:58.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 547956 13.50 MB/sec execute 105 sec latency 111.069 ms 2024-09-21T01:33:59.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 163835 3.85 MB/sec execute 108 sec latency 24.074 ms 2024-09-21T01:33:59.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 551416 13.55 MB/sec execute 106 sec latency 86.920 ms 2024-09-21T01:34:00.050 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:33:59 smithi090 ceph-mon[35142]: pgmap v386: 129 pgs: 129 active+clean; 82 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:34:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:33:59 smithi045 ceph-mon[28361]: pgmap v386: 129 pgs: 129 active+clean; 82 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:34:00.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 164593 3.86 MB/sec execute 109 sec latency 15.788 ms 2024-09-21T01:34:00.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 551423 13.43 MB/sec execute 107 sec latency 291.785 ms 2024-09-21T01:34:01.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 165366 3.83 MB/sec execute 110 sec latency 25.534 ms 2024-09-21T01:34:01.560 INFO:tasks.workunit.client.0.smithi045.stdout: 1 553533 13.41 MB/sec execute 108 sec latency 275.152 ms 2024-09-21T01:34:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:01 smithi045 ceph-mon[28361]: pgmap v387: 129 pgs: 129 active+clean; 83 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:34:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:01 smithi090 ceph-mon[35142]: pgmap v387: 129 pgs: 129 active+clean; 83 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:34:02.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 166192 3.86 MB/sec execute 111 sec latency 17.853 ms 2024-09-21T01:34:02.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 556538 13.41 MB/sec execute 109 sec latency 34.486 ms 2024-09-21T01:34:03.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 166939 3.88 MB/sec execute 112 sec latency 14.427 ms 2024-09-21T01:34:03.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 559595 13.45 MB/sec execute 110 sec latency 41.732 ms 2024-09-21T01:34:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:03 smithi045 ceph-mon[28361]: pgmap v388: 129 pgs: 129 active+clean; 83 MiB data, 718 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:34:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:03 smithi090 ceph-mon[35142]: pgmap v388: 129 pgs: 129 active+clean; 83 MiB data, 718 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:34:04.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 167427 3.85 MB/sec execute 113 sec latency 17.484 ms 2024-09-21T01:34:04.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 562246 13.46 MB/sec execute 111 sec latency 49.540 ms 2024-09-21T01:34:05.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 168221 3.86 MB/sec execute 114 sec latency 22.050 ms 2024-09-21T01:34:05.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 565607 13.51 MB/sec execute 112 sec latency 33.514 ms 2024-09-21T01:34:06.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:05 smithi045 ceph-mon[28361]: pgmap v389: 129 pgs: 129 active+clean; 83 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:34:06.200 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:05 smithi090 ceph-mon[35142]: pgmap v389: 129 pgs: 129 active+clean; 83 MiB data, 723 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:34:06.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 169041 3.84 MB/sec execute 115 sec latency 15.900 ms 2024-09-21T01:34:06.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 569023 13.56 MB/sec execute 113 sec latency 177.125 ms 2024-09-21T01:34:07.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 169987 3.87 MB/sec execute 116 sec latency 42.160 ms 2024-09-21T01:34:07.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 569033 13.44 MB/sec execute 114 sec latency 308.388 ms 2024-09-21T01:34:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:07 smithi045 ceph-mon[28361]: pgmap v390: 129 pgs: 129 active+clean; 85 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:34:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:07 smithi090 ceph-mon[35142]: pgmap v390: 129 pgs: 129 active+clean; 85 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:34:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:08.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 170518 3.88 MB/sec execute 117 sec latency 14.593 ms 2024-09-21T01:34:08.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 571644 13.44 MB/sec execute 115 sec latency 158.505 ms 2024-09-21T01:34:09.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 171026 3.85 MB/sec execute 118 sec latency 17.695 ms 2024-09-21T01:34:09.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 574806 13.48 MB/sec execute 116 sec latency 41.709 ms 2024-09-21T01:34:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:09 smithi045 ceph-mon[28361]: pgmap v391: 129 pgs: 129 active+clean; 84 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-21T01:34:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:09 smithi090 ceph-mon[35142]: pgmap v391: 129 pgs: 129 active+clean; 84 MiB data, 694 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-21T01:34:10.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 171770 3.86 MB/sec execute 119 sec latency 13.643 ms 2024-09-21T01:34:10.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 577852 13.48 MB/sec execute 117 sec latency 33.241 ms 2024-09-21T01:34:11.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 172596 3.84 MB/sec execute 120 sec latency 16.557 ms 2024-09-21T01:34:11.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 580465 13.51 MB/sec execute 118 sec latency 99.846 ms 2024-09-21T01:34:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:11 smithi045 ceph-mon[28361]: pgmap v392: 129 pgs: 129 active+clean; 84 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 90 op/s 2024-09-21T01:34:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:11 smithi090 ceph-mon[35142]: pgmap v392: 129 pgs: 129 active+clean; 84 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 90 op/s 2024-09-21T01:34:12.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 173538 3.87 MB/sec execute 121 sec latency 13.392 ms 2024-09-21T01:34:12.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 583450 13.52 MB/sec execute 119 sec latency 34.469 ms 2024-09-21T01:34:13.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 174038 3.87 MB/sec execute 122 sec latency 90.519 ms 2024-09-21T01:34:13.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 585296 13.49 MB/sec execute 120 sec latency 291.121 ms 2024-09-21T01:34:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:13 smithi045 ceph-mon[28361]: pgmap v393: 129 pgs: 129 active+clean; 84 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:34:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:13 smithi090 ceph-mon[35142]: pgmap v393: 129 pgs: 129 active+clean; 84 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:34:14.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 174488 3.85 MB/sec execute 123 sec latency 49.902 ms 2024-09-21T01:34:14.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 585855 13.41 MB/sec execute 121 sec latency 316.685 ms 2024-09-21T01:34:15.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 175294 3.85 MB/sec execute 124 sec latency 12.237 ms 2024-09-21T01:34:15.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 589070 13.44 MB/sec execute 122 sec latency 50.074 ms 2024-09-21T01:34:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:15 smithi045 ceph-mon[28361]: pgmap v394: 129 pgs: 129 active+clean; 84 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-21T01:34:16.201 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:15 smithi090 ceph-mon[35142]: pgmap v394: 129 pgs: 129 active+clean; 84 MiB data, 696 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-21T01:34:16.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 176129 3.84 MB/sec execute 125 sec latency 13.077 ms 2024-09-21T01:34:16.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 591999 13.44 MB/sec execute 123 sec latency 91.782 ms 2024-09-21T01:34:17.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 177090 3.87 MB/sec execute 126 sec latency 15.084 ms 2024-09-21T01:34:17.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 594804 13.47 MB/sec execute 124 sec latency 50.086 ms 2024-09-21T01:34:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:17 smithi045 ceph-mon[28361]: pgmap v395: 129 pgs: 129 active+clean; 87 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:34:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:17 smithi090 ceph-mon[35142]: pgmap v395: 129 pgs: 129 active+clean; 87 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:34:18.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 177625 3.87 MB/sec execute 127 sec latency 14.745 ms 2024-09-21T01:34:18.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 597701 13.48 MB/sec execute 125 sec latency 35.189 ms 2024-09-21T01:34:19.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 178189 3.85 MB/sec execute 128 sec latency 17.131 ms 2024-09-21T01:34:19.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 601062 13.53 MB/sec execute 126 sec latency 34.716 ms 2024-09-21T01:34:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:19 smithi045 ceph-mon[28361]: pgmap v396: 129 pgs: 129 active+clean; 87 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:34:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:19 smithi090 ceph-mon[35142]: pgmap v396: 129 pgs: 129 active+clean; 87 MiB data, 706 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:34:20.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 178902 3.85 MB/sec execute 129 sec latency 15.827 ms 2024-09-21T01:34:20.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 601139 13.42 MB/sec execute 127 sec latency 300.150 ms 2024-09-21T01:34:21.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 179704 3.84 MB/sec execute 130 sec latency 44.263 ms 2024-09-21T01:34:21.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 603251 13.41 MB/sec execute 128 sec latency 283.480 ms 2024-09-21T01:34:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:21 smithi045 ceph-mon[28361]: pgmap v397: 129 pgs: 129 active+clean; 88 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2024-09-21T01:34:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:21 smithi090 ceph-mon[35142]: pgmap v397: 129 pgs: 129 active+clean; 88 MiB data, 697 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 97 op/s 2024-09-21T01:34:22.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 180648 3.87 MB/sec execute 131 sec latency 14.074 ms 2024-09-21T01:34:22.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 606222 13.41 MB/sec execute 129 sec latency 50.108 ms 2024-09-21T01:34:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:23.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 181197 3.87 MB/sec execute 132 sec latency 12.777 ms 2024-09-21T01:34:23.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 608271 13.41 MB/sec execute 130 sec latency 127.551 ms 2024-09-21T01:34:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:23 smithi045 ceph-mon[28361]: pgmap v398: 129 pgs: 129 active+clean; 87 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:34:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:23 smithi090 ceph-mon[35142]: pgmap v398: 129 pgs: 129 active+clean; 87 MiB data, 702 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:34:24.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 181737 3.85 MB/sec execute 133 sec latency 21.647 ms 2024-09-21T01:34:24.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 611702 13.45 MB/sec execute 131 sec latency 41.760 ms 2024-09-21T01:34:25.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 182524 3.85 MB/sec execute 134 sec latency 11.547 ms 2024-09-21T01:34:25.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 615181 13.49 MB/sec execute 132 sec latency 52.526 ms 2024-09-21T01:34:26.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:25 smithi045 ceph-mon[28361]: pgmap v399: 129 pgs: 129 active+clean; 88 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-21T01:34:26.201 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:25 smithi090 ceph-mon[35142]: pgmap v399: 129 pgs: 129 active+clean; 88 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-21T01:34:26.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 183310 3.84 MB/sec execute 135 sec latency 20.706 ms 2024-09-21T01:34:26.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 618669 13.53 MB/sec execute 133 sec latency 81.598 ms 2024-09-21T01:34:27.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 184217 3.86 MB/sec execute 136 sec latency 13.132 ms 2024-09-21T01:34:27.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 621311 13.54 MB/sec execute 134 sec latency 55.415 ms 2024-09-21T01:34:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:27 smithi045 ceph-mon[28361]: pgmap v400: 129 pgs: 129 active+clean; 88 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-21T01:34:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:27 smithi090 ceph-mon[35142]: pgmap v400: 129 pgs: 129 active+clean; 88 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-21T01:34:28.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 184796 3.87 MB/sec execute 137 sec latency 14.130 ms 2024-09-21T01:34:28.561 INFO:tasks.workunit.client.0.smithi045.stdout: 1 624005 13.53 MB/sec execute 135 sec latency 94.087 ms 2024-09-21T01:34:29.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 185384 3.85 MB/sec execute 138 sec latency 17.141 ms 2024-09-21T01:34:29.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 626716 13.55 MB/sec execute 136 sec latency 35.243 ms 2024-09-21T01:34:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:29 smithi045 ceph-mon[28361]: pgmap v401: 129 pgs: 129 active+clean; 90 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:34:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:29 smithi090 ceph-mon[35142]: pgmap v401: 129 pgs: 129 active+clean; 90 MiB data, 698 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:34:30.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 186153 3.85 MB/sec execute 139 sec latency 18.009 ms 2024-09-21T01:34:30.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 629540 13.56 MB/sec execute 137 sec latency 90.687 ms 2024-09-21T01:34:31.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 186921 3.84 MB/sec execute 140 sec latency 14.980 ms 2024-09-21T01:34:31.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 632974 13.60 MB/sec execute 138 sec latency 61.028 ms 2024-09-21T01:34:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:31 smithi045 ceph-mon[28361]: pgmap v402: 129 pgs: 129 active+clean; 90 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-09-21T01:34:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:31 smithi090 ceph-mon[35142]: pgmap v402: 129 pgs: 129 active+clean; 90 MiB data, 704 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 88 op/s 2024-09-21T01:34:32.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 187772 3.86 MB/sec execute 141 sec latency 13.668 ms 2024-09-21T01:34:32.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 636454 13.64 MB/sec execute 139 sec latency 33.209 ms 2024-09-21T01:34:33.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 188393 3.87 MB/sec execute 142 sec latency 13.660 ms 2024-09-21T01:34:33.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 639938 13.67 MB/sec execute 140 sec latency 91.368 ms 2024-09-21T01:34:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:33 smithi045 ceph-mon[28361]: pgmap v403: 129 pgs: 129 active+clean; 89 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-21T01:34:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:33 smithi090 ceph-mon[35142]: pgmap v403: 129 pgs: 129 active+clean; 89 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-21T01:34:34.501 INFO:tasks.workunit.client.1.smithi090.stdout: 1 189043 3.85 MB/sec execute 143 sec latency 15.366 ms 2024-09-21T01:34:34.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 642559 13.68 MB/sec execute 141 sec latency 33.441 ms 2024-09-21T01:34:35.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 189850 3.85 MB/sec execute 144 sec latency 16.944 ms 2024-09-21T01:34:35.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 645779 13.70 MB/sec execute 142 sec latency 33.396 ms 2024-09-21T01:34:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:35 smithi045 ceph-mon[28361]: pgmap v404: 129 pgs: 129 active+clean; 91 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:34:36.201 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:35 smithi090 ceph-mon[35142]: pgmap v404: 129 pgs: 129 active+clean; 91 MiB data, 728 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:34:36.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 190602 3.84 MB/sec execute 145 sec latency 15.255 ms 2024-09-21T01:34:36.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 648697 13.70 MB/sec execute 143 sec latency 45.136 ms 2024-09-21T01:34:37.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 191393 3.86 MB/sec execute 146 sec latency 11.396 ms 2024-09-21T01:34:37.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 651529 13.73 MB/sec execute 144 sec latency 50.169 ms 2024-09-21T01:34:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:37 smithi045 ceph-mon[28361]: pgmap v405: 129 pgs: 129 active+clean; 91 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:34:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:37 smithi090 ceph-mon[35142]: pgmap v405: 129 pgs: 129 active+clean; 91 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:34:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:38.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 192025 3.87 MB/sec execute 147 sec latency 12.422 ms 2024-09-21T01:34:38.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 654364 13.73 MB/sec execute 145 sec latency 38.310 ms 2024-09-21T01:34:39.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 192711 3.85 MB/sec execute 148 sec latency 16.348 ms 2024-09-21T01:34:39.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 657792 13.77 MB/sec execute 146 sec latency 33.634 ms 2024-09-21T01:34:40.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:39 smithi045 ceph-mon[28361]: pgmap v406: 129 pgs: 129 active+clean; 91 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:34:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:39 smithi090 ceph-mon[35142]: pgmap v406: 129 pgs: 129 active+clean; 91 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:34:40.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 193528 3.85 MB/sec execute 149 sec latency 14.773 ms 2024-09-21T01:34:40.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 661271 13.80 MB/sec execute 147 sec latency 41.785 ms 2024-09-21T01:34:41.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 194325 3.86 MB/sec execute 150 sec latency 17.094 ms 2024-09-21T01:34:41.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 664811 13.84 MB/sec execute 148 sec latency 37.201 ms 2024-09-21T01:34:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:41 smithi045 ceph-mon[28361]: pgmap v407: 129 pgs: 129 active+clean; 93 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:34:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:41 smithi090 ceph-mon[35142]: pgmap v407: 129 pgs: 129 active+clean; 93 MiB data, 752 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:34:42.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 195106 3.88 MB/sec execute 151 sec latency 14.244 ms 2024-09-21T01:34:42.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 666551 13.80 MB/sec execute 149 sec latency 241.892 ms 2024-09-21T01:34:43.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 195664 3.87 MB/sec execute 152 sec latency 12.446 ms 2024-09-21T01:34:43.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 669513 13.82 MB/sec execute 150 sec latency 41.774 ms 2024-09-21T01:34:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:43 smithi045 ceph-mon[28361]: pgmap v408: 129 pgs: 129 active+clean; 95 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:34:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:43 smithi090 ceph-mon[35142]: pgmap v408: 129 pgs: 129 active+clean; 95 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:34:44.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 196347 3.85 MB/sec execute 153 sec latency 24.007 ms 2024-09-21T01:34:44.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 672155 13.82 MB/sec execute 151 sec latency 41.670 ms 2024-09-21T01:34:45.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 197135 3.85 MB/sec execute 154 sec latency 96.409 ms 2024-09-21T01:34:45.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 675650 13.86 MB/sec execute 152 sec latency 44.137 ms 2024-09-21T01:34:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:45 smithi045 ceph-mon[28361]: pgmap v409: 129 pgs: 129 active+clean; 96 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 86 op/s 2024-09-21T01:34:46.201 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:45 smithi090 ceph-mon[35142]: pgmap v409: 129 pgs: 129 active+clean; 96 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 86 op/s 2024-09-21T01:34:46.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 197995 3.87 MB/sec execute 155 sec latency 17.157 ms 2024-09-21T01:34:46.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 679112 13.89 MB/sec execute 153 sec latency 33.409 ms 2024-09-21T01:34:47.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 198725 3.88 MB/sec execute 156 sec latency 14.477 ms 2024-09-21T01:34:47.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 681075 13.87 MB/sec execute 154 sec latency 249.248 ms 2024-09-21T01:34:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:47 smithi045 ceph-mon[28361]: pgmap v410: 129 pgs: 129 active+clean; 95 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:34:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:47 smithi090 ceph-mon[35142]: pgmap v410: 129 pgs: 129 active+clean; 95 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:34:48.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 199160 3.86 MB/sec execute 157 sec latency 85.250 ms 2024-09-21T01:34:48.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 681306 13.79 MB/sec execute 155 sec latency 308.273 ms 2024-09-21T01:34:49.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 199809 3.85 MB/sec execute 158 sec latency 41.471 ms 2024-09-21T01:34:49.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 681635 13.71 MB/sec execute 156 sec latency 316.823 ms 2024-09-21T01:34:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:49 smithi045 ceph-mon[28361]: pgmap v411: 129 pgs: 129 active+clean; 94 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:34:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:49 smithi090 ceph-mon[35142]: pgmap v411: 129 pgs: 129 active+clean; 94 MiB data, 760 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:34:50.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 200607 3.85 MB/sec execute 159 sec latency 17.491 ms 2024-09-21T01:34:50.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 682536 13.65 MB/sec execute 157 sec latency 275.019 ms 2024-09-21T01:34:51.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 201383 3.86 MB/sec execute 160 sec latency 33.408 ms 2024-09-21T01:34:51.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 682546 13.57 MB/sec execute 158 sec latency 291.821 ms 2024-09-21T01:34:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:51 smithi045 ceph-mon[28361]: pgmap v412: 129 pgs: 129 active+clean; 96 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-21T01:34:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:51 smithi090 ceph-mon[35142]: pgmap v412: 129 pgs: 129 active+clean; 96 MiB data, 779 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-21T01:34:52.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 201969 3.86 MB/sec execute 161 sec latency 71.650 ms 2024-09-21T01:34:52.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 682595 13.48 MB/sec execute 159 sec latency 299.960 ms 2024-09-21T01:34:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:34:53.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 202589 3.86 MB/sec execute 162 sec latency 52.422 ms 2024-09-21T01:34:53.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 682658 13.40 MB/sec execute 160 sec latency 283.512 ms 2024-09-21T01:34:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:53 smithi045 ceph-mon[28361]: pgmap v413: 129 pgs: 129 active+clean; 96 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:34:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:53 smithi090 ceph-mon[35142]: pgmap v413: 129 pgs: 129 active+clean; 96 MiB data, 783 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:34:54.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 203181 3.85 MB/sec execute 163 sec latency 50.778 ms 2024-09-21T01:34:54.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 682667 13.32 MB/sec execute 161 sec latency 325.256 ms 2024-09-21T01:34:55.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 203975 3.85 MB/sec execute 164 sec latency 15.524 ms 2024-09-21T01:34:55.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 686141 13.35 MB/sec execute 162 sec latency 58.596 ms 2024-09-21T01:34:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:55 smithi045 ceph-mon[28361]: pgmap v414: 129 pgs: 129 active+clean; 96 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-21T01:34:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:34:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:34:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:34:56.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:55 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:34:56.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:55 smithi090 ceph-mon[35142]: pgmap v414: 129 pgs: 129 active+clean; 96 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-21T01:34:56.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:34:56.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:34:56.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:34:56.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:55 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:34:56.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 204681 3.84 MB/sec execute 165 sec latency 22.655 ms 2024-09-21T01:34:56.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 689683 13.39 MB/sec execute 163 sec latency 33.216 ms 2024-09-21T01:34:57.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 205511 3.86 MB/sec execute 166 sec latency 14.020 ms 2024-09-21T01:34:57.562 INFO:tasks.workunit.client.0.smithi045.stdout: 1 693177 13.42 MB/sec execute 164 sec latency 33.432 ms 2024-09-21T01:34:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:57 smithi045 ceph-mon[28361]: pgmap v415: 129 pgs: 129 active+clean; 97 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 103 op/s 2024-09-21T01:34:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:57 smithi090 ceph-mon[35142]: pgmap v415: 129 pgs: 129 active+clean; 97 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 103 op/s 2024-09-21T01:34:58.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 206156 3.86 MB/sec execute 167 sec latency 12.356 ms 2024-09-21T01:34:58.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 695818 13.43 MB/sec execute 165 sec latency 64.095 ms 2024-09-21T01:34:59.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 206823 3.85 MB/sec execute 168 sec latency 16.842 ms 2024-09-21T01:34:59.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 699019 13.45 MB/sec execute 166 sec latency 85.991 ms 2024-09-21T01:35:00.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:34:59 smithi045 ceph-mon[28361]: pgmap v416: 129 pgs: 129 active+clean; 98 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-21T01:35:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:34:59 smithi090 ceph-mon[35142]: pgmap v416: 129 pgs: 129 active+clean; 98 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-21T01:35:00.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 207557 3.85 MB/sec execute 169 sec latency 87.840 ms 2024-09-21T01:35:00.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 700327 13.41 MB/sec execute 167 sec latency 258.501 ms 2024-09-21T01:35:01.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 208230 3.84 MB/sec execute 170 sec latency 38.659 ms 2024-09-21T01:35:01.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 700506 13.33 MB/sec execute 168 sec latency 319.571 ms 2024-09-21T01:35:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:01 smithi045 ceph-mon[28361]: pgmap v417: 129 pgs: 129 active+clean; 100 MiB data, 801 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-21T01:35:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:01 smithi090 ceph-mon[35142]: pgmap v417: 129 pgs: 129 active+clean; 100 MiB data, 801 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-21T01:35:02.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 209055 3.86 MB/sec execute 171 sec latency 12.533 ms 2024-09-21T01:35:02.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 703933 13.37 MB/sec execute 169 sec latency 33.653 ms 2024-09-21T01:35:03.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 209676 3.86 MB/sec execute 172 sec latency 12.817 ms 2024-09-21T01:35:03.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 707408 13.40 MB/sec execute 170 sec latency 33.410 ms 2024-09-21T01:35:03.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:03 smithi045 ceph-mon[28361]: pgmap v418: 129 pgs: 129 active+clean; 99 MiB data, 800 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-21T01:35:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:03 smithi090 ceph-mon[35142]: pgmap v418: 129 pgs: 129 active+clean; 99 MiB data, 800 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-21T01:35:04.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 210315 3.85 MB/sec execute 173 sec latency 16.381 ms 2024-09-21T01:35:04.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 709999 13.40 MB/sec execute 171 sec latency 88.571 ms 2024-09-21T01:35:05.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:04 smithi045 ceph-mon[28361]: pgmap v419: 129 pgs: 129 active+clean; 98 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-21T01:35:05.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:04 smithi090 ceph-mon[35142]: pgmap v419: 129 pgs: 129 active+clean; 98 MiB data, 798 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-21T01:35:05.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 211106 3.85 MB/sec execute 174 sec latency 12.947 ms 2024-09-21T01:35:05.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 712989 13.42 MB/sec execute 172 sec latency 75.836 ms 2024-09-21T01:35:06.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 211829 3.84 MB/sec execute 175 sec latency 22.199 ms 2024-09-21T01:35:06.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 716112 13.43 MB/sec execute 173 sec latency 33.280 ms 2024-09-21T01:35:07.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:07 smithi090 ceph-mon[35142]: pgmap v420: 129 pgs: 129 active+clean; 99 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:35:07.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 212600 3.85 MB/sec execute 176 sec latency 31.591 ms 2024-09-21T01:35:07.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 716745 13.38 MB/sec execute 174 sec latency 273.390 ms 2024-09-21T01:35:07.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:07 smithi045 ceph-mon[28361]: pgmap v420: 129 pgs: 129 active+clean; 99 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:35:08.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:08.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 213215 3.86 MB/sec execute 177 sec latency 47.796 ms 2024-09-21T01:35:08.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 718058 13.34 MB/sec execute 175 sec latency 291.841 ms 2024-09-21T01:35:08.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:09.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:09 smithi090 ceph-mon[35142]: pgmap v421: 129 pgs: 129 active+clean; 99 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:35:09.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 213861 3.85 MB/sec execute 178 sec latency 16.857 ms 2024-09-21T01:35:09.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 721591 13.38 MB/sec execute 176 sec latency 42.228 ms 2024-09-21T01:35:09.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:09 smithi045 ceph-mon[28361]: pgmap v421: 129 pgs: 129 active+clean; 99 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:35:10.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 214658 3.85 MB/sec execute 179 sec latency 13.530 ms 2024-09-21T01:35:10.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 725087 13.41 MB/sec execute 177 sec latency 49.093 ms 2024-09-21T01:35:11.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:11 smithi090 ceph-mon[35142]: pgmap v422: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-09-21T01:35:11.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 215434 3.84 MB/sec execute 180 sec latency 14.997 ms 2024-09-21T01:35:11.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 727706 13.41 MB/sec execute 178 sec latency 108.361 ms 2024-09-21T01:35:11.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:11 smithi045 ceph-mon[28361]: pgmap v422: 129 pgs: 129 active+clean; 101 MiB data, 832 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-09-21T01:35:12.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 216215 3.85 MB/sec execute 181 sec latency 12.496 ms 2024-09-21T01:35:12.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 730944 13.43 MB/sec execute 179 sec latency 33.331 ms 2024-09-21T01:35:13.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:13 smithi090 ceph-mon[35142]: pgmap v423: 129 pgs: 129 active+clean; 102 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:13.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 216890 3.86 MB/sec execute 182 sec latency 9.826 ms 2024-09-21T01:35:13.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 733974 13.44 MB/sec execute 180 sec latency 108.542 ms 2024-09-21T01:35:13.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:13 smithi045 ceph-mon[28361]: pgmap v423: 129 pgs: 129 active+clean; 102 MiB data, 834 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:14.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 217553 3.85 MB/sec execute 183 sec latency 16.399 ms 2024-09-21T01:35:14.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 736931 13.46 MB/sec execute 181 sec latency 33.575 ms 2024-09-21T01:35:15.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:15 smithi090 ceph-mon[35142]: pgmap v424: 129 pgs: 129 active+clean; 102 MiB data, 837 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:35:15.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 218342 3.85 MB/sec execute 184 sec latency 20.629 ms 2024-09-21T01:35:15.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 740074 13.49 MB/sec execute 182 sec latency 50.091 ms 2024-09-21T01:35:15.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:15 smithi045 ceph-mon[28361]: pgmap v424: 129 pgs: 129 active+clean; 102 MiB data, 837 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:35:16.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 219181 3.86 MB/sec execute 185 sec latency 16.613 ms 2024-09-21T01:35:16.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 743052 13.49 MB/sec execute 183 sec latency 37.062 ms 2024-09-21T01:35:17.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:17 smithi090 ceph-mon[35142]: pgmap v425: 129 pgs: 129 active+clean; 103 MiB data, 842 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:35:17.502 INFO:tasks.workunit.client.1.smithi090.stdout: 1 219977 3.87 MB/sec execute 186 sec latency 10.364 ms 2024-09-21T01:35:17.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 746486 13.52 MB/sec execute 184 sec latency 30.301 ms 2024-09-21T01:35:17.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:17 smithi045 ceph-mon[28361]: pgmap v425: 129 pgs: 129 active+clean; 103 MiB data, 842 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:35:18.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 220505 3.86 MB/sec execute 187 sec latency 10.537 ms 2024-09-21T01:35:18.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 749957 13.55 MB/sec execute 185 sec latency 100.271 ms 2024-09-21T01:35:19.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:19 smithi090 ceph-mon[35142]: pgmap v426: 129 pgs: 129 active+clean; 101 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:35:19.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 221197 3.85 MB/sec execute 188 sec latency 16.442 ms 2024-09-21T01:35:19.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 753456 13.58 MB/sec execute 186 sec latency 33.271 ms 2024-09-21T01:35:19.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:19 smithi045 ceph-mon[28361]: pgmap v426: 129 pgs: 129 active+clean; 101 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:35:20.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 221944 3.85 MB/sec execute 189 sec latency 110.901 ms 2024-09-21T01:35:20.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 756077 13.59 MB/sec execute 187 sec latency 118.749 ms 2024-09-21T01:35:21.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:21 smithi090 ceph-mon[35142]: pgmap v427: 129 pgs: 129 active+clean; 103 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-09-21T01:35:21.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 222723 3.86 MB/sec execute 190 sec latency 14.800 ms 2024-09-21T01:35:21.563 INFO:tasks.workunit.client.0.smithi045.stdout: 1 759238 13.60 MB/sec execute 188 sec latency 36.193 ms 2024-09-21T01:35:21.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:21 smithi045 ceph-mon[28361]: pgmap v427: 129 pgs: 129 active+clean; 103 MiB data, 868 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.7 MiB/s wr, 96 op/s 2024-09-21T01:35:22.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 223477 3.87 MB/sec execute 191 sec latency 13.615 ms 2024-09-21T01:35:22.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 761504 13.60 MB/sec execute 189 sec latency 45.872 ms 2024-09-21T01:35:23.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:23 smithi090 ceph-mon[35142]: pgmap v428: 129 pgs: 129 active+clean; 104 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:23.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:23.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 224039 3.86 MB/sec execute 192 sec latency 12.171 ms 2024-09-21T01:35:23.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 764293 13.60 MB/sec execute 190 sec latency 50.056 ms 2024-09-21T01:35:23.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:23 smithi045 ceph-mon[28361]: pgmap v428: 129 pgs: 129 active+clean; 104 MiB data, 874 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:23.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:24.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 224751 3.85 MB/sec execute 193 sec latency 15.872 ms 2024-09-21T01:35:24.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 767696 13.63 MB/sec execute 191 sec latency 91.429 ms 2024-09-21T01:35:25.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:25 smithi090 ceph-mon[35142]: pgmap v429: 129 pgs: 129 active+clean; 104 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:35:25.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 225568 3.85 MB/sec execute 194 sec latency 15.352 ms 2024-09-21T01:35:25.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 771234 13.66 MB/sec execute 192 sec latency 35.045 ms 2024-09-21T01:35:25.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:25 smithi045 ceph-mon[28361]: pgmap v429: 129 pgs: 129 active+clean; 104 MiB data, 881 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:35:26.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 226428 3.86 MB/sec execute 195 sec latency 14.684 ms 2024-09-21T01:35:26.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 774725 13.68 MB/sec execute 193 sec latency 53.909 ms 2024-09-21T01:35:27.471 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:27 smithi045 ceph-mon[28361]: pgmap v430: 129 pgs: 129 active+clean; 105 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-21T01:35:27.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:27 smithi090 ceph-mon[35142]: pgmap v430: 129 pgs: 129 active+clean; 105 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-21T01:35:27.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 227083 3.87 MB/sec execute 196 sec latency 15.028 ms 2024-09-21T01:35:27.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 777242 13.68 MB/sec execute 194 sec latency 68.921 ms 2024-09-21T01:35:28.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 227644 3.86 MB/sec execute 197 sec latency 14.112 ms 2024-09-21T01:35:28.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 780008 13.68 MB/sec execute 195 sec latency 41.949 ms 2024-09-21T01:35:29.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:29 smithi090 ceph-mon[35142]: pgmap v431: 129 pgs: 129 active+clean; 107 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:35:29.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 228311 3.85 MB/sec execute 198 sec latency 64.727 ms 2024-09-21T01:35:29.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 781951 13.67 MB/sec execute 196 sec latency 166.799 ms 2024-09-21T01:35:29.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:29 smithi045 ceph-mon[28361]: pgmap v431: 129 pgs: 129 active+clean; 107 MiB data, 888 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:35:30.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 229152 3.85 MB/sec execute 199 sec latency 13.075 ms 2024-09-21T01:35:30.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 785426 13.70 MB/sec execute 197 sec latency 91.504 ms 2024-09-21T01:35:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:31 smithi090 ceph-mon[35142]: pgmap v432: 129 pgs: 129 active+clean; 106 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:35:31.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 230019 3.86 MB/sec execute 200 sec latency 12.659 ms 2024-09-21T01:35:31.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 788964 13.72 MB/sec execute 198 sec latency 33.409 ms 2024-09-21T01:35:31.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:31 smithi045 ceph-mon[28361]: pgmap v432: 129 pgs: 129 active+clean; 106 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-21T01:35:32.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 230708 3.87 MB/sec execute 201 sec latency 13.158 ms 2024-09-21T01:35:32.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 792449 13.75 MB/sec execute 199 sec latency 125.194 ms 2024-09-21T01:35:33.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:33 smithi090 ceph-mon[35142]: pgmap v433: 129 pgs: 129 active+clean; 105 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 74 op/s 2024-09-21T01:35:33.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 231218 3.86 MB/sec execute 202 sec latency 15.085 ms 2024-09-21T01:35:33.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 794966 13.75 MB/sec execute 200 sec latency 33.443 ms 2024-09-21T01:35:33.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:33 smithi045 ceph-mon[28361]: pgmap v433: 129 pgs: 129 active+clean; 105 MiB data, 894 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 74 op/s 2024-09-21T01:35:34.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 231992 3.86 MB/sec execute 203 sec latency 19.724 ms 2024-09-21T01:35:34.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 796313 13.71 MB/sec execute 201 sec latency 184.689 ms 2024-09-21T01:35:35.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:35 smithi090 ceph-mon[35142]: pgmap v434: 129 pgs: 129 active+clean; 106 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:35:35.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 232703 3.85 MB/sec execute 204 sec latency 47.985 ms 2024-09-21T01:35:35.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 797212 13.67 MB/sec execute 202 sec latency 333.573 ms 2024-09-21T01:35:35.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:35 smithi045 ceph-mon[28361]: pgmap v434: 129 pgs: 129 active+clean; 106 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:35:36.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 233386 3.85 MB/sec execute 205 sec latency 50.378 ms 2024-09-21T01:35:36.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 797668 13.61 MB/sec execute 203 sec latency 332.090 ms 2024-09-21T01:35:37.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:37 smithi090 ceph-mon[35142]: pgmap v435: 129 pgs: 129 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:35:37.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 234138 3.87 MB/sec execute 206 sec latency 52.616 ms 2024-09-21T01:35:37.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 797800 13.55 MB/sec execute 204 sec latency 324.973 ms 2024-09-21T01:35:37.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:37 smithi045 ceph-mon[28361]: pgmap v435: 129 pgs: 129 active+clean; 107 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:35:38.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:38.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 234662 3.85 MB/sec execute 207 sec latency 15.814 ms 2024-09-21T01:35:38.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 798318 13.50 MB/sec execute 205 sec latency 299.966 ms 2024-09-21T01:35:38.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:39.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:39 smithi090 ceph-mon[35142]: pgmap v436: 129 pgs: 129 active+clean; 108 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:35:39.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 235242 3.85 MB/sec execute 208 sec latency 80.242 ms 2024-09-21T01:35:39.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 798622 13.45 MB/sec execute 206 sec latency 339.848 ms 2024-09-21T01:35:39.617 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:39 smithi045 ceph-mon[28361]: pgmap v436: 129 pgs: 129 active+clean; 108 MiB data, 895 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:35:40.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 235967 3.84 MB/sec execute 209 sec latency 40.933 ms 2024-09-21T01:35:40.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 799541 13.40 MB/sec execute 207 sec latency 316.642 ms 2024-09-21T01:35:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:41 smithi090 ceph-mon[35142]: pgmap v437: 129 pgs: 129 active+clean; 109 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:35:41.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 236698 3.84 MB/sec execute 210 sec latency 43.388 ms 2024-09-21T01:35:41.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 799548 13.34 MB/sec execute 208 sec latency 325.231 ms 2024-09-21T01:35:41.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:41 smithi045 ceph-mon[28361]: pgmap v437: 129 pgs: 129 active+clean; 109 MiB data, 897 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:35:42.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 237481 3.85 MB/sec execute 211 sec latency 15.450 ms 2024-09-21T01:35:42.564 INFO:tasks.workunit.client.0.smithi045.stdout: 1 800756 13.30 MB/sec execute 209 sec latency 266.835 ms 2024-09-21T01:35:43.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:43 smithi090 ceph-mon[35142]: pgmap v438: 129 pgs: 129 active+clean; 108 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:43.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 238147 3.85 MB/sec execute 212 sec latency 13.744 ms 2024-09-21T01:35:43.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 803404 13.31 MB/sec execute 210 sec latency 41.778 ms 2024-09-21T01:35:43.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:43 smithi045 ceph-mon[28361]: pgmap v438: 129 pgs: 129 active+clean; 108 MiB data, 896 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:44.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 238805 3.85 MB/sec execute 213 sec latency 22.487 ms 2024-09-21T01:35:44.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 806841 13.33 MB/sec execute 211 sec latency 49.521 ms 2024-09-21T01:35:45.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:45 smithi090 ceph-mon[35142]: pgmap v439: 129 pgs: 129 active+clean; 110 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 92 op/s 2024-09-21T01:35:45.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 239605 3.84 MB/sec execute 214 sec latency 13.468 ms 2024-09-21T01:35:45.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 810307 13.36 MB/sec execute 212 sec latency 116.824 ms 2024-09-21T01:35:45.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:45 smithi045 ceph-mon[28361]: pgmap v439: 129 pgs: 129 active+clean; 110 MiB data, 899 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 92 op/s 2024-09-21T01:35:46.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 240413 3.85 MB/sec execute 215 sec latency 18.831 ms 2024-09-21T01:35:46.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 813789 13.39 MB/sec execute 213 sec latency 50.020 ms 2024-09-21T01:35:47.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:47 smithi090 ceph-mon[35142]: pgmap v440: 129 pgs: 129 active+clean; 109 MiB data, 902 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-21T01:35:47.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 241219 3.86 MB/sec execute 216 sec latency 23.249 ms 2024-09-21T01:35:47.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 815523 13.36 MB/sec execute 214 sec latency 308.154 ms 2024-09-21T01:35:47.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:47 smithi045 ceph-mon[28361]: pgmap v440: 129 pgs: 129 active+clean; 109 MiB data, 902 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-21T01:35:48.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 241713 3.85 MB/sec execute 217 sec latency 65.811 ms 2024-09-21T01:35:48.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 816024 13.32 MB/sec execute 215 sec latency 324.059 ms 2024-09-21T01:35:49.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:49 smithi090 ceph-mon[35142]: pgmap v441: 129 pgs: 129 active+clean; 109 MiB data, 906 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:35:49.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 242393 3.85 MB/sec execute 218 sec latency 31.897 ms 2024-09-21T01:35:49.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 818486 13.31 MB/sec execute 216 sec latency 224.883 ms 2024-09-21T01:35:49.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:49 smithi045 ceph-mon[28361]: pgmap v441: 129 pgs: 129 active+clean; 109 MiB data, 906 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:35:50.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 243155 3.84 MB/sec execute 219 sec latency 12.950 ms 2024-09-21T01:35:50.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 821125 13.32 MB/sec execute 217 sec latency 50.363 ms 2024-09-21T01:35:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:51 smithi090 ceph-mon[35142]: pgmap v442: 129 pgs: 129 active+clean; 110 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:35:51.503 INFO:tasks.workunit.client.1.smithi090.stdout: 1 243989 3.85 MB/sec execute 220 sec latency 14.360 ms 2024-09-21T01:35:51.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 824567 13.34 MB/sec execute 218 sec latency 33.417 ms 2024-09-21T01:35:51.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:51 smithi045 ceph-mon[28361]: pgmap v442: 129 pgs: 129 active+clean; 110 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:35:52.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 244758 3.86 MB/sec execute 221 sec latency 12.963 ms 2024-09-21T01:35:52.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 828028 13.37 MB/sec execute 219 sec latency 75.151 ms 2024-09-21T01:35:53.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:53 smithi090 ceph-mon[35142]: pgmap v443: 129 pgs: 129 active+clean; 111 MiB data, 910 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:53.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:53.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 245298 3.85 MB/sec execute 222 sec latency 13.542 ms 2024-09-21T01:35:53.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 831508 13.40 MB/sec execute 220 sec latency 49.324 ms 2024-09-21T01:35:53.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:53 smithi045 ceph-mon[28361]: pgmap v443: 129 pgs: 129 active+clean; 111 MiB data, 910 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:35:53.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:35:54.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 246019 3.85 MB/sec execute 223 sec latency 16.385 ms 2024-09-21T01:35:54.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 835000 13.42 MB/sec execute 221 sec latency 40.266 ms 2024-09-21T01:35:55.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:55 smithi045 ceph-mon[28361]: pgmap v444: 129 pgs: 129 active+clean; 112 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:35:55.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:55 smithi090 ceph-mon[35142]: pgmap v444: 129 pgs: 129 active+clean; 112 MiB data, 916 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:35:55.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 246847 3.84 MB/sec execute 224 sec latency 19.039 ms 2024-09-21T01:35:55.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 837660 13.42 MB/sec execute 222 sec latency 50.062 ms 2024-09-21T01:35:56.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 247687 3.85 MB/sec execute 225 sec latency 19.016 ms 2024-09-21T01:35:56.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 840886 13.44 MB/sec execute 223 sec latency 35.673 ms 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:57 smithi090 ceph-mon[35142]: pgmap v445: 129 pgs: 129 active+clean; 113 MiB data, 925 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:35:57.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:35:57.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:35:57.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:35:57.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:35:57.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 248352 3.86 MB/sec execute 226 sec latency 15.611 ms 2024-09-21T01:35:57.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 843850 13.44 MB/sec execute 224 sec latency 41.955 ms 2024-09-21T01:35:57.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:57 smithi045 ceph-mon[28361]: pgmap v445: 129 pgs: 129 active+clean; 113 MiB data, 925 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:35:57.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:35:57.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:35:57.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:35:57.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:35:58.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 248920 3.85 MB/sec execute 227 sec latency 15.401 ms 2024-09-21T01:35:58.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 846508 13.46 MB/sec execute 225 sec latency 75.595 ms 2024-09-21T01:35:59.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:35:59 smithi090 ceph-mon[35142]: pgmap v446: 129 pgs: 129 active+clean; 112 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:35:59.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 249620 3.84 MB/sec execute 228 sec latency 15.889 ms 2024-09-21T01:35:59.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 849447 13.46 MB/sec execute 226 sec latency 58.685 ms 2024-09-21T01:35:59.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:35:59 smithi045 ceph-mon[28361]: pgmap v446: 129 pgs: 129 active+clean; 112 MiB data, 932 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:36:00.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 250472 3.84 MB/sec execute 229 sec latency 13.122 ms 2024-09-21T01:36:00.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 852913 13.49 MB/sec execute 227 sec latency 74.129 ms 2024-09-21T01:36:01.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:01 smithi090 ceph-mon[35142]: pgmap v447: 129 pgs: 129 active+clean; 113 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:36:01.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 251320 3.86 MB/sec execute 230 sec latency 13.132 ms 2024-09-21T01:36:01.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 856395 13.51 MB/sec execute 228 sec latency 27.563 ms 2024-09-21T01:36:01.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:01 smithi045 ceph-mon[28361]: pgmap v447: 129 pgs: 129 active+clean; 113 MiB data, 933 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:36:02.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 251995 3.86 MB/sec execute 231 sec latency 14.378 ms 2024-09-21T01:36:02.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 859940 13.54 MB/sec execute 229 sec latency 33.642 ms 2024-09-21T01:36:03.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:03 smithi090 ceph-mon[35142]: pgmap v448: 129 pgs: 129 active+clean; 114 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:36:03.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 252506 3.85 MB/sec execute 232 sec latency 15.618 ms 2024-09-21T01:36:03.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 863428 13.56 MB/sec execute 230 sec latency 40.015 ms 2024-09-21T01:36:03.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:03 smithi045 ceph-mon[28361]: pgmap v448: 129 pgs: 129 active+clean; 114 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:36:04.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 253312 3.86 MB/sec execute 233 sec latency 11.973 ms 2024-09-21T01:36:04.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 866081 13.56 MB/sec execute 231 sec latency 50.079 ms 2024-09-21T01:36:05.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 254103 3.84 MB/sec execute 234 sec latency 19.952 ms 2024-09-21T01:36:05.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 869059 13.57 MB/sec execute 232 sec latency 100.107 ms 2024-09-21T01:36:05.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:05 smithi045 ceph-mon[28361]: pgmap v449: 129 pgs: 129 active+clean; 114 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:36:05.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:05 smithi090 ceph-mon[35142]: pgmap v449: 129 pgs: 129 active+clean; 114 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:36:06.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 255080 3.86 MB/sec execute 235 sec latency 13.310 ms 2024-09-21T01:36:06.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 871728 13.57 MB/sec execute 233 sec latency 87.231 ms 2024-09-21T01:36:07.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 255678 3.87 MB/sec execute 236 sec latency 11.446 ms 2024-09-21T01:36:07.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 874872 13.60 MB/sec execute 234 sec latency 33.736 ms 2024-09-21T01:36:07.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:07 smithi045 ceph-mon[28361]: pgmap v450: 129 pgs: 129 active+clean; 113 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-09-21T01:36:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:07 smithi090 ceph-mon[35142]: pgmap v450: 129 pgs: 129 active+clean; 113 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-09-21T01:36:08.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 256236 3.85 MB/sec execute 237 sec latency 15.807 ms 2024-09-21T01:36:08.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 877814 13.60 MB/sec execute 235 sec latency 33.451 ms 2024-09-21T01:36:08.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:08.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:09.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 257016 3.86 MB/sec execute 238 sec latency 11.607 ms 2024-09-21T01:36:09.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 881157 13.62 MB/sec execute 236 sec latency 266.241 ms 2024-09-21T01:36:09.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:09 smithi045 ceph-mon[28361]: pgmap v451: 129 pgs: 129 active+clean; 114 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:36:09.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:09 smithi090 ceph-mon[35142]: pgmap v451: 129 pgs: 129 active+clean; 114 MiB data, 953 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:36:10.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 257814 3.85 MB/sec execute 239 sec latency 21.438 ms 2024-09-21T01:36:10.565 INFO:tasks.workunit.client.0.smithi045.stdout: 1 883242 13.61 MB/sec execute 237 sec latency 205.735 ms 2024-09-21T01:36:11.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:11 smithi090 ceph-mon[35142]: pgmap v452: 129 pgs: 129 active+clean; 114 MiB data, 957 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:36:11.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 258728 3.86 MB/sec execute 240 sec latency 12.769 ms 2024-09-21T01:36:11.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 886327 13.61 MB/sec execute 238 sec latency 33.697 ms 2024-09-21T01:36:11.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:11 smithi045 ceph-mon[28361]: pgmap v452: 129 pgs: 129 active+clean; 114 MiB data, 957 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:36:12.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 259324 3.87 MB/sec execute 241 sec latency 13.878 ms 2024-09-21T01:36:12.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 888559 13.61 MB/sec execute 239 sec latency 99.928 ms 2024-09-21T01:36:13.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 259903 3.85 MB/sec execute 242 sec latency 17.067 ms 2024-09-21T01:36:13.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 891993 13.64 MB/sec execute 240 sec latency 41.457 ms 2024-09-21T01:36:13.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:13 smithi045 ceph-mon[28361]: pgmap v453: 129 pgs: 129 active+clean; 115 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-21T01:36:13.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:13 smithi090 ceph-mon[35142]: pgmap v453: 129 pgs: 129 active+clean; 115 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-21T01:36:14.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 260666 3.86 MB/sec execute 243 sec latency 12.993 ms 2024-09-21T01:36:14.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 895457 13.66 MB/sec execute 241 sec latency 41.457 ms 2024-09-21T01:36:15.504 INFO:tasks.workunit.client.1.smithi090.stdout: 1 261364 3.85 MB/sec execute 244 sec latency 98.652 ms 2024-09-21T01:36:15.551 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:15 smithi045 ceph-mon[28361]: pgmap v454: 129 pgs: 129 active+clean; 115 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 77 op/s 2024-09-21T01:36:15.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 895584 13.60 MB/sec execute 242 sec latency 301.667 ms 2024-09-21T01:36:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:15 smithi090 ceph-mon[35142]: pgmap v454: 129 pgs: 129 active+clean; 115 MiB data, 961 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 77 op/s 2024-09-21T01:36:16.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 262249 3.86 MB/sec execute 245 sec latency 73.077 ms 2024-09-21T01:36:16.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 896295 13.56 MB/sec execute 243 sec latency 308.516 ms 2024-09-21T01:36:17.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 262723 3.86 MB/sec execute 246 sec latency 53.595 ms 2024-09-21T01:36:17.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 896549 13.51 MB/sec execute 244 sec latency 333.508 ms 2024-09-21T01:36:17.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:17 smithi045 ceph-mon[28361]: pgmap v455: 129 pgs: 129 active+clean; 117 MiB data, 963 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:36:17.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:17 smithi090 ceph-mon[35142]: pgmap v455: 129 pgs: 129 active+clean; 117 MiB data, 963 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:36:18.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 263125 3.85 MB/sec execute 247 sec latency 86.925 ms 2024-09-21T01:36:18.706 INFO:tasks.workunit.client.0.smithi045.stdout: 1 897035 13.46 MB/sec execute 245 sec latency 363.827 ms 2024-09-21T01:36:19.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 263744 3.84 MB/sec execute 248 sec latency 60.500 ms 2024-09-21T01:36:19.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 897577 13.43 MB/sec execute 246 sec latency 340.400 ms 2024-09-21T01:36:19.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:19 smithi045 ceph-mon[28361]: pgmap v456: 129 pgs: 129 active+clean; 117 MiB data, 963 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:36:19.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:19 smithi090 ceph-mon[35142]: pgmap v456: 129 pgs: 129 active+clean; 117 MiB data, 963 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:36:20.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 264514 3.84 MB/sec execute 249 sec latency 55.027 ms 2024-09-21T01:36:20.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 897852 13.38 MB/sec execute 247 sec latency 308.692 ms 2024-09-21T01:36:21.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:21 smithi090 ceph-mon[35142]: pgmap v457: 129 pgs: 129 active+clean; 118 MiB data, 963 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-09-21T01:36:21.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 265256 3.85 MB/sec execute 250 sec latency 48.346 ms 2024-09-21T01:36:21.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 897983 13.33 MB/sec execute 248 sec latency 307.603 ms 2024-09-21T01:36:21.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:21 smithi045 ceph-mon[28361]: pgmap v457: 129 pgs: 129 active+clean; 118 MiB data, 963 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 91 op/s 2024-09-21T01:36:22.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 265903 3.85 MB/sec execute 251 sec latency 14.408 ms 2024-09-21T01:36:22.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 900617 13.32 MB/sec execute 249 sec latency 266.630 ms 2024-09-21T01:36:23.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 266523 3.85 MB/sec execute 252 sec latency 21.040 ms 2024-09-21T01:36:23.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 903136 13.34 MB/sec execute 250 sec latency 76.946 ms 2024-09-21T01:36:23.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:23 smithi045 ceph-mon[28361]: pgmap v458: 129 pgs: 129 active+clean; 118 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:36:23.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:23 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:23.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:23 smithi090 ceph-mon[35142]: pgmap v458: 129 pgs: 129 active+clean; 118 MiB data, 974 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:36:23.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:23 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:24.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 267192 3.84 MB/sec execute 253 sec latency 21.891 ms 2024-09-21T01:36:24.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 906177 13.34 MB/sec execute 251 sec latency 33.459 ms 2024-09-21T01:36:25.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 267986 3.84 MB/sec execute 254 sec latency 18.960 ms 2024-09-21T01:36:25.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 909641 13.36 MB/sec execute 252 sec latency 41.740 ms 2024-09-21T01:36:25.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:25 smithi045 ceph-mon[28361]: pgmap v459: 129 pgs: 129 active+clean; 118 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:36:25.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:25 smithi090 ceph-mon[35142]: pgmap v459: 129 pgs: 129 active+clean; 118 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:36:26.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 268805 3.85 MB/sec execute 255 sec latency 12.768 ms 2024-09-21T01:36:26.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 913119 13.39 MB/sec execute 253 sec latency 41.606 ms 2024-09-21T01:36:27.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 269584 3.86 MB/sec execute 256 sec latency 12.813 ms 2024-09-21T01:36:27.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 914272 13.35 MB/sec execute 254 sec latency 266.868 ms 2024-09-21T01:36:27.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:27 smithi045 ceph-mon[28361]: pgmap v460: 129 pgs: 129 active+clean; 120 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-21T01:36:27.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:27 smithi090 ceph-mon[35142]: pgmap v460: 129 pgs: 129 active+clean; 120 MiB data, 966 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-21T01:36:28.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 270054 3.85 MB/sec execute 257 sec latency 75.268 ms 2024-09-21T01:36:28.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 914729 13.31 MB/sec execute 255 sec latency 308.540 ms 2024-09-21T01:36:29.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 270706 3.84 MB/sec execute 258 sec latency 17.132 ms 2024-09-21T01:36:29.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 917484 13.39 MB/sec execute 256 sec latency 289.654 ms 2024-09-21T01:36:29.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:29 smithi045 ceph-mon[28361]: pgmap v461: 129 pgs: 129 active+clean; 120 MiB data, 971 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:36:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:29 smithi090 ceph-mon[35142]: pgmap v461: 129 pgs: 129 active+clean; 120 MiB data, 971 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:36:30.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 271467 3.84 MB/sec execute 259 sec latency 14.287 ms 2024-09-21T01:36:30.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 920989 13.41 MB/sec execute 257 sec latency 34.617 ms 2024-09-21T01:36:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:31 smithi090 ceph-mon[35142]: pgmap v462: 129 pgs: 129 active+clean; 122 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:36:31.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 272198 3.84 MB/sec execute 260 sec latency 20.364 ms 2024-09-21T01:36:31.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 924466 13.43 MB/sec execute 258 sec latency 116.614 ms 2024-09-21T01:36:31.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:31 smithi045 ceph-mon[28361]: pgmap v462: 129 pgs: 129 active+clean; 122 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:36:32.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 272973 3.85 MB/sec execute 261 sec latency 13.709 ms 2024-09-21T01:36:32.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 926984 13.43 MB/sec execute 259 sec latency 50.151 ms 2024-09-21T01:36:33.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 273623 3.85 MB/sec execute 262 sec latency 12.464 ms 2024-09-21T01:36:33.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 930107 13.44 MB/sec execute 260 sec latency 33.367 ms 2024-09-21T01:36:33.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:33 smithi045 ceph-mon[28361]: pgmap v463: 129 pgs: 129 active+clean; 122 MiB data, 997 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:36:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:33 smithi090 ceph-mon[35142]: pgmap v463: 129 pgs: 129 active+clean; 122 MiB data, 997 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:36:34.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 274223 3.84 MB/sec execute 263 sec latency 72.227 ms 2024-09-21T01:36:34.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 930666 13.40 MB/sec execute 261 sec latency 274.586 ms 2024-09-21T01:36:35.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 275020 3.84 MB/sec execute 264 sec latency 34.576 ms 2024-09-21T01:36:35.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 931807 13.37 MB/sec execute 262 sec latency 291.659 ms 2024-09-21T01:36:35.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:35 smithi045 ceph-mon[28361]: pgmap v464: 129 pgs: 129 active+clean; 123 MiB data, 999 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-21T01:36:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:35 smithi090 ceph-mon[35142]: pgmap v464: 129 pgs: 129 active+clean; 123 MiB data, 999 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-21T01:36:36.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 275734 3.83 MB/sec execute 265 sec latency 20.227 ms 2024-09-21T01:36:36.566 INFO:tasks.workunit.client.0.smithi045.stdout: 1 935169 13.39 MB/sec execute 263 sec latency 191.736 ms 2024-09-21T01:36:37.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 276527 3.84 MB/sec execute 266 sec latency 13.424 ms 2024-09-21T01:36:37.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 938698 13.42 MB/sec execute 264 sec latency 33.219 ms 2024-09-21T01:36:37.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:37 smithi045 ceph-mon[28361]: pgmap v465: 129 pgs: 129 active+clean; 123 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T01:36:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:37 smithi090 ceph-mon[35142]: pgmap v465: 129 pgs: 129 active+clean; 123 MiB data, 1002 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T01:36:38.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 277173 3.85 MB/sec execute 267 sec latency 12.217 ms 2024-09-21T01:36:38.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 941350 13.42 MB/sec execute 265 sec latency 33.428 ms 2024-09-21T01:36:38.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:38.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:39.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 277858 3.84 MB/sec execute 268 sec latency 17.539 ms 2024-09-21T01:36:39.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 943980 13.41 MB/sec execute 266 sec latency 125.066 ms 2024-09-21T01:36:39.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:39 smithi045 ceph-mon[28361]: pgmap v466: 129 pgs: 129 active+clean; 123 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-09-21T01:36:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:39 smithi090 ceph-mon[35142]: pgmap v466: 129 pgs: 129 active+clean; 123 MiB data, 992 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 81 op/s 2024-09-21T01:36:40.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 278674 3.84 MB/sec execute 269 sec latency 20.968 ms 2024-09-21T01:36:40.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 945917 13.41 MB/sec execute 267 sec latency 133.475 ms 2024-09-21T01:36:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:41 smithi090 ceph-mon[35142]: pgmap v467: 129 pgs: 129 active+clean; 124 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:36:41.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 279452 3.85 MB/sec execute 270 sec latency 26.612 ms 2024-09-21T01:36:41.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 946028 13.36 MB/sec execute 268 sec latency 308.599 ms 2024-09-21T01:36:41.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:41 smithi045 ceph-mon[28361]: pgmap v467: 129 pgs: 129 active+clean; 124 MiB data, 993 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:36:42.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 280245 3.86 MB/sec execute 271 sec latency 14.520 ms 2024-09-21T01:36:42.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 948400 13.36 MB/sec execute 269 sec latency 303.667 ms 2024-09-21T01:36:43.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 280755 3.85 MB/sec execute 272 sec latency 13.631 ms 2024-09-21T01:36:43.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 951131 13.36 MB/sec execute 270 sec latency 83.591 ms 2024-09-21T01:36:43.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:43 smithi045 ceph-mon[28361]: pgmap v468: 129 pgs: 129 active+clean; 126 MiB data, 999 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:36:43.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:43 smithi090 ceph-mon[35142]: pgmap v468: 129 pgs: 129 active+clean; 126 MiB data, 999 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:36:44.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 281430 3.84 MB/sec execute 273 sec latency 18.436 ms 2024-09-21T01:36:44.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 953836 13.37 MB/sec execute 271 sec latency 41.729 ms 2024-09-21T01:36:45.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 282298 3.84 MB/sec execute 274 sec latency 14.992 ms 2024-09-21T01:36:45.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 957232 13.39 MB/sec execute 272 sec latency 33.785 ms 2024-09-21T01:36:45.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:45 smithi045 ceph-mon[28361]: pgmap v469: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:36:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:45 smithi090 ceph-mon[35142]: pgmap v469: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:36:46.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 283142 3.85 MB/sec execute 275 sec latency 19.257 ms 2024-09-21T01:36:46.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 960169 13.40 MB/sec execute 273 sec latency 108.437 ms 2024-09-21T01:36:47.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 283806 3.85 MB/sec execute 276 sec latency 16.804 ms 2024-09-21T01:36:47.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 963636 13.42 MB/sec execute 274 sec latency 74.091 ms 2024-09-21T01:36:47.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:47 smithi045 ceph-mon[28361]: pgmap v470: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:36:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:47 smithi090 ceph-mon[35142]: pgmap v470: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:36:48.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 284353 3.85 MB/sec execute 277 sec latency 10.181 ms 2024-09-21T01:36:48.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 963643 13.37 MB/sec execute 275 sec latency 316.945 ms 2024-09-21T01:36:49.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 285024 3.84 MB/sec execute 278 sec latency 38.582 ms 2024-09-21T01:36:49.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 965600 13.36 MB/sec execute 276 sec latency 325.138 ms 2024-09-21T01:36:49.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:49 smithi045 ceph-mon[28361]: pgmap v471: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-21T01:36:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:49 smithi090 ceph-mon[35142]: pgmap v471: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-21T01:36:50.505 INFO:tasks.workunit.client.1.smithi090.stdout: 1 285849 3.84 MB/sec execute 279 sec latency 15.681 ms 2024-09-21T01:36:50.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 968756 13.36 MB/sec execute 277 sec latency 33.361 ms 2024-09-21T01:36:51.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 286700 3.85 MB/sec execute 280 sec latency 16.299 ms 2024-09-21T01:36:51.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 971564 13.38 MB/sec execute 278 sec latency 54.091 ms 2024-09-21T01:36:51.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:51 smithi045 ceph-mon[28361]: pgmap v472: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:36:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:51 smithi090 ceph-mon[35142]: pgmap v472: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:36:52.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 287418 3.86 MB/sec execute 281 sec latency 14.351 ms 2024-09-21T01:36:52.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 974395 13.38 MB/sec execute 279 sec latency 65.433 ms 2024-09-21T01:36:52.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:36:53.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 287953 3.85 MB/sec execute 282 sec latency 10.264 ms 2024-09-21T01:36:53.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 977829 13.40 MB/sec execute 280 sec latency 35.149 ms 2024-09-21T01:36:53.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:53 smithi045 ceph-mon[28361]: pgmap v473: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:36:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:53 smithi090 ceph-mon[35142]: pgmap v473: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:36:54.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 288736 3.85 MB/sec execute 283 sec latency 16.885 ms 2024-09-21T01:36:54.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 981363 13.42 MB/sec execute 281 sec latency 49.885 ms 2024-09-21T01:36:55.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 289517 3.84 MB/sec execute 284 sec latency 13.204 ms 2024-09-21T01:36:55.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 984850 13.44 MB/sec execute 282 sec latency 43.310 ms 2024-09-21T01:36:55.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:55 smithi045 ceph-mon[28361]: pgmap v474: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:36:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:55 smithi090 ceph-mon[35142]: pgmap v474: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:36:56.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 290498 3.85 MB/sec execute 285 sec latency 14.649 ms 2024-09-21T01:36:56.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 988383 13.46 MB/sec execute 283 sec latency 32.672 ms 2024-09-21T01:36:57.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 291022 3.86 MB/sec execute 286 sec latency 12.986 ms 2024-09-21T01:36:58.149 INFO:tasks.workunit.client.0.smithi045.stdout: 1 991910 13.48 MB/sec execute 284 sec latency 45.042 ms 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: pgmap v475: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:36:58.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:57 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: pgmap v475: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:36:58.150 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:57 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:36:58.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 291541 3.85 MB/sec execute 287 sec latency 17.035 ms 2024-09-21T01:36:58.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 994526 13.48 MB/sec execute 285 sec latency 81.829 ms 2024-09-21T01:36:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:58 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi045 to 913.0M 2024-09-21T01:36:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:58 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi045 to 913.0M 2024-09-21T01:36:59.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 292330 3.85 MB/sec execute 288 sec latency 12.302 ms 2024-09-21T01:36:59.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 997192 13.48 MB/sec execute 286 sec latency 116.926 ms 2024-09-21T01:36:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:36:59 smithi090 ceph-mon[35142]: pgmap v476: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-21T01:37:00.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:36:59 smithi045 ceph-mon[28361]: pgmap v476: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-21T01:37:00.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 293169 3.84 MB/sec execute 289 sec latency 14.212 ms 2024-09-21T01:37:00.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 999961 13.49 MB/sec execute 287 sec latency 116.702 ms 2024-09-21T01:37:01.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 294118 3.85 MB/sec execute 290 sec latency 14.042 ms 2024-09-21T01:37:01.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1002788 13.50 MB/sec execute 288 sec latency 33.757 ms 2024-09-21T01:37:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:01 smithi090 ceph-mon[35142]: pgmap v477: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:37:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:01 smithi045 ceph-mon[28361]: pgmap v477: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:37:02.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 294682 3.86 MB/sec execute 291 sec latency 13.405 ms 2024-09-21T01:37:02.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1006102 13.51 MB/sec execute 289 sec latency 232.946 ms 2024-09-21T01:37:03.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 295248 3.85 MB/sec execute 292 sec latency 15.773 ms 2024-09-21T01:37:03.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1009636 13.53 MB/sec execute 290 sec latency 33.124 ms 2024-09-21T01:37:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:03 smithi090 ceph-mon[35142]: pgmap v478: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-21T01:37:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:03 smithi045 ceph-mon[28361]: pgmap v478: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-21T01:37:04.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 295986 3.85 MB/sec execute 293 sec latency 13.197 ms 2024-09-21T01:37:04.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1011305 13.51 MB/sec execute 291 sec latency 142.029 ms 2024-09-21T01:37:05.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 296774 3.84 MB/sec execute 294 sec latency 51.421 ms 2024-09-21T01:37:05.567 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1013428 13.51 MB/sec execute 292 sec latency 255.734 ms 2024-09-21T01:37:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:05 smithi090 ceph-mon[35142]: pgmap v479: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:37:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:05 smithi045 ceph-mon[28361]: pgmap v479: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:37:06.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 297719 3.86 MB/sec execute 295 sec latency 10.341 ms 2024-09-21T01:37:06.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1016859 13.53 MB/sec execute 293 sec latency 33.074 ms 2024-09-21T01:37:07.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 298281 3.86 MB/sec execute 296 sec latency 12.790 ms 2024-09-21T01:37:07.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1020335 13.54 MB/sec execute 294 sec latency 110.918 ms 2024-09-21T01:37:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:07 smithi090 ceph-mon[35142]: pgmap v480: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:37:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:07 smithi045 ceph-mon[28361]: pgmap v480: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:37:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:08.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 298845 3.85 MB/sec execute 297 sec latency 28.768 ms 2024-09-21T01:37:08.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1023817 13.56 MB/sec execute 295 sec latency 175.099 ms 2024-09-21T01:37:09.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 299578 3.85 MB/sec execute 298 sec latency 19.063 ms 2024-09-21T01:37:09.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1026435 13.56 MB/sec execute 296 sec latency 62.420 ms 2024-09-21T01:37:09.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:09 smithi090 ceph-mon[35142]: pgmap v481: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-21T01:37:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:09 smithi045 ceph-mon[28361]: pgmap v481: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-21T01:37:10.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 300337 3.84 MB/sec execute 299 sec latency 33.448 ms 2024-09-21T01:37:10.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1027496 13.54 MB/sec execute 297 sec latency 300.375 ms 2024-09-21T01:37:11.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 301253 3.86 MB/sec execute 300 sec latency 72.280 ms 2024-09-21T01:37:11.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1027575 13.49 MB/sec execute 298 sec latency 316.716 ms 2024-09-21T01:37:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:11 smithi090 ceph-mon[35142]: pgmap v482: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:37:12.068 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:11 smithi045 ceph-mon[28361]: pgmap v482: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-21T01:37:12.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 301775 3.86 MB/sec execute 301 sec latency 77.808 ms 2024-09-21T01:37:12.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1027681 13.45 MB/sec execute 299 sec latency 300.209 ms 2024-09-21T01:37:13.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 302274 3.85 MB/sec execute 302 sec latency 63.127 ms 2024-09-21T01:37:13.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1028328 13.42 MB/sec execute 300 sec latency 283.442 ms 2024-09-21T01:37:13.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:13 smithi090 ceph-mon[35142]: pgmap v483: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-21T01:37:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:13 smithi045 ceph-mon[28361]: pgmap v483: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-21T01:37:14.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 302993 3.85 MB/sec execute 303 sec latency 85.292 ms 2024-09-21T01:37:14.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1029034 13.38 MB/sec execute 301 sec latency 316.945 ms 2024-09-21T01:37:15.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 303698 3.84 MB/sec execute 304 sec latency 93.038 ms 2024-09-21T01:37:15.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1029166 13.34 MB/sec execute 302 sec latency 308.325 ms 2024-09-21T01:37:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:15 smithi090 ceph-mon[35142]: pgmap v484: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:37:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:15 smithi045 ceph-mon[28361]: pgmap v484: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:37:16.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 304515 3.85 MB/sec execute 305 sec latency 49.238 ms 2024-09-21T01:37:16.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1031034 13.33 MB/sec execute 303 sec latency 315.126 ms 2024-09-21T01:37:17.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 305229 3.85 MB/sec execute 306 sec latency 12.617 ms 2024-09-21T01:37:17.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1034510 13.35 MB/sec execute 304 sec latency 80.781 ms 2024-09-21T01:37:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:17 smithi090 ceph-mon[35142]: pgmap v485: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:37:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:17 smithi045 ceph-mon[28361]: pgmap v485: 129 pgs: 129 active+clean; 133 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:37:18.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 305732 3.85 MB/sec execute 307 sec latency 18.313 ms 2024-09-21T01:37:18.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1038000 13.37 MB/sec execute 305 sec latency 33.432 ms 2024-09-21T01:37:19.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 306520 3.85 MB/sec execute 308 sec latency 12.354 ms 2024-09-21T01:37:19.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1040623 13.37 MB/sec execute 306 sec latency 50.091 ms 2024-09-21T01:37:20.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:19 smithi090 ceph-mon[35142]: pgmap v486: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:37:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:19 smithi045 ceph-mon[28361]: pgmap v486: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:37:20.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 307270 3.84 MB/sec execute 309 sec latency 17.767 ms 2024-09-21T01:37:20.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1043353 13.37 MB/sec execute 307 sec latency 105.808 ms 2024-09-21T01:37:21.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 308250 3.85 MB/sec execute 310 sec latency 12.360 ms 2024-09-21T01:37:21.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1045339 13.36 MB/sec execute 308 sec latency 184.358 ms 2024-09-21T01:37:21.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:21 smithi090 ceph-mon[35142]: pgmap v487: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-09-21T01:37:22.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:21 smithi045 ceph-mon[28361]: pgmap v487: 129 pgs: 129 active+clean; 132 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-09-21T01:37:22.506 INFO:tasks.workunit.client.1.smithi090.stdout: 1 308843 3.86 MB/sec execute 311 sec latency 48.217 ms 2024-09-21T01:37:22.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1045903 13.33 MB/sec execute 309 sec latency 333.084 ms 2024-09-21T01:37:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:23.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 309269 3.84 MB/sec execute 312 sec latency 83.765 ms 2024-09-21T01:37:23.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1048706 13.34 MB/sec execute 310 sec latency 255.837 ms 2024-09-21T01:37:23.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:23 smithi090 ceph-mon[35142]: pgmap v488: 129 pgs: 129 active+clean; 134 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:37:24.183 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:23 smithi045 ceph-mon[28361]: pgmap v488: 129 pgs: 129 active+clean; 134 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:37:24.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 310042 3.85 MB/sec execute 313 sec latency 15.115 ms 2024-09-21T01:37:24.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1052197 13.36 MB/sec execute 311 sec latency 33.487 ms 2024-09-21T01:37:25.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 310839 3.84 MB/sec execute 314 sec latency 19.101 ms 2024-09-21T01:37:25.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1054845 13.36 MB/sec execute 312 sec latency 33.482 ms 2024-09-21T01:37:25.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:25 smithi090 ceph-mon[35142]: pgmap v489: 129 pgs: 129 active+clean; 134 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:37:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:25 smithi045 ceph-mon[28361]: pgmap v489: 129 pgs: 129 active+clean; 134 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:37:26.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 311816 3.85 MB/sec execute 315 sec latency 12.631 ms 2024-09-21T01:37:26.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1058251 13.37 MB/sec execute 313 sec latency 33.462 ms 2024-09-21T01:37:27.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 312412 3.86 MB/sec execute 316 sec latency 13.227 ms 2024-09-21T01:37:27.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1061077 13.37 MB/sec execute 314 sec latency 33.430 ms 2024-09-21T01:37:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:27 smithi045 ceph-mon[28361]: pgmap v490: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-21T01:37:27.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:27 smithi090 ceph-mon[35142]: pgmap v490: 129 pgs: 129 active+clean; 136 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-21T01:37:28.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 312979 3.85 MB/sec execute 317 sec latency 12.873 ms 2024-09-21T01:37:28.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1063140 13.37 MB/sec execute 315 sec latency 58.750 ms 2024-09-21T01:37:29.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 313749 3.85 MB/sec execute 318 sec latency 12.332 ms 2024-09-21T01:37:29.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1066612 13.39 MB/sec execute 316 sec latency 41.673 ms 2024-09-21T01:37:29.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:29 smithi090 ceph-mon[35142]: pgmap v491: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:37:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:29 smithi045 ceph-mon[28361]: pgmap v491: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 80 op/s 2024-09-21T01:37:30.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 314548 3.84 MB/sec execute 319 sec latency 14.844 ms 2024-09-21T01:37:30.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1070046 13.41 MB/sec execute 317 sec latency 41.929 ms 2024-09-21T01:37:31.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 315456 3.85 MB/sec execute 320 sec latency 12.617 ms 2024-09-21T01:37:31.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1073522 13.42 MB/sec execute 318 sec latency 41.651 ms 2024-09-21T01:37:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:31 smithi090 ceph-mon[35142]: pgmap v492: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:37:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:31 smithi045 ceph-mon[28361]: pgmap v492: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:37:32.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 316042 3.86 MB/sec execute 321 sec latency 10.685 ms 2024-09-21T01:37:32.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1077016 13.44 MB/sec execute 319 sec latency 33.452 ms 2024-09-21T01:37:33.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 316642 3.85 MB/sec execute 322 sec latency 19.607 ms 2024-09-21T01:37:33.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1079327 13.44 MB/sec execute 320 sec latency 162.251 ms 2024-09-21T01:37:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:33 smithi090 ceph-mon[35142]: pgmap v493: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:37:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:33 smithi045 ceph-mon[28361]: pgmap v493: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:37:34.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 317417 3.85 MB/sec execute 323 sec latency 11.140 ms 2024-09-21T01:37:34.568 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1082636 13.45 MB/sec execute 321 sec latency 25.150 ms 2024-09-21T01:37:35.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 318172 3.84 MB/sec execute 324 sec latency 22.414 ms 2024-09-21T01:37:35.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1085060 13.45 MB/sec execute 322 sec latency 116.381 ms 2024-09-21T01:37:35.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:35 smithi090 ceph-mon[35142]: pgmap v494: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-21T01:37:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:35 smithi045 ceph-mon[28361]: pgmap v494: 129 pgs: 129 active+clean; 136 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-21T01:37:36.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 319020 3.85 MB/sec execute 325 sec latency 13.434 ms 2024-09-21T01:37:36.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1087950 13.45 MB/sec execute 323 sec latency 26.992 ms 2024-09-21T01:37:37.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 319638 3.86 MB/sec execute 326 sec latency 13.858 ms 2024-09-21T01:37:37.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1091323 13.47 MB/sec execute 324 sec latency 50.138 ms 2024-09-21T01:37:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:37 smithi090 ceph-mon[35142]: pgmap v495: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-21T01:37:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:37 smithi045 ceph-mon[28361]: pgmap v495: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-21T01:37:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:38.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 320285 3.85 MB/sec execute 327 sec latency 15.923 ms 2024-09-21T01:37:38.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1094795 13.49 MB/sec execute 325 sec latency 88.024 ms 2024-09-21T01:37:39.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 321066 3.85 MB/sec execute 328 sec latency 15.574 ms 2024-09-21T01:37:39.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1098289 13.50 MB/sec execute 326 sec latency 33.662 ms 2024-09-21T01:37:39.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:39 smithi090 ceph-mon[35142]: pgmap v496: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:37:40.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:39 smithi045 ceph-mon[28361]: pgmap v496: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:37:40.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 321795 3.84 MB/sec execute 329 sec latency 22.086 ms 2024-09-21T01:37:40.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1101822 13.52 MB/sec execute 327 sec latency 35.572 ms 2024-09-21T01:37:41.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 322579 3.85 MB/sec execute 330 sec latency 13.623 ms 2024-09-21T01:37:41.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1103910 13.51 MB/sec execute 328 sec latency 116.763 ms 2024-09-21T01:37:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:41 smithi090 ceph-mon[35142]: pgmap v497: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:37:42.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:41 smithi045 ceph-mon[28361]: pgmap v497: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:37:42.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 323229 3.85 MB/sec execute 331 sec latency 12.641 ms 2024-09-21T01:37:42.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1106586 13.51 MB/sec execute 329 sec latency 41.716 ms 2024-09-21T01:37:43.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 323853 3.85 MB/sec execute 332 sec latency 19.490 ms 2024-09-21T01:37:43.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1109037 13.51 MB/sec execute 330 sec latency 265.031 ms 2024-09-21T01:37:43.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:43 smithi090 ceph-mon[35142]: pgmap v498: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:37:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:43 smithi045 ceph-mon[28361]: pgmap v498: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-21T01:37:44.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 324642 3.85 MB/sec execute 333 sec latency 15.195 ms 2024-09-21T01:37:44.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1112519 13.53 MB/sec execute 331 sec latency 266.778 ms 2024-09-21T01:37:45.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 325365 3.84 MB/sec execute 334 sec latency 20.362 ms 2024-09-21T01:37:45.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1116010 13.55 MB/sec execute 332 sec latency 31.053 ms 2024-09-21T01:37:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:45 smithi090 ceph-mon[35142]: pgmap v499: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:37:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:45 smithi045 ceph-mon[28361]: pgmap v499: 129 pgs: 129 active+clean; 139 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:37:46.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 326134 3.85 MB/sec execute 335 sec latency 13.676 ms 2024-09-21T01:37:46.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1118663 13.55 MB/sec execute 333 sec latency 40.405 ms 2024-09-21T01:37:47.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 326777 3.85 MB/sec execute 336 sec latency 13.956 ms 2024-09-21T01:37:47.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1121855 13.56 MB/sec execute 334 sec latency 34.519 ms 2024-09-21T01:37:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:47 smithi090 ceph-mon[35142]: pgmap v500: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:37:48.057 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:47 smithi045 ceph-mon[28361]: pgmap v500: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-21T01:37:48.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 327430 3.85 MB/sec execute 337 sec latency 18.081 ms 2024-09-21T01:37:48.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1124763 13.56 MB/sec execute 335 sec latency 33.455 ms 2024-09-21T01:37:49.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 328221 3.85 MB/sec execute 338 sec latency 15.627 ms 2024-09-21T01:37:49.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1126652 13.56 MB/sec execute 336 sec latency 291.795 ms 2024-09-21T01:37:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:49 smithi090 ceph-mon[35142]: pgmap v501: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:37:50.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:49 smithi045 ceph-mon[28361]: pgmap v501: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 78 op/s 2024-09-21T01:37:50.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 328919 3.84 MB/sec execute 339 sec latency 48.625 ms 2024-09-21T01:37:50.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1126701 13.52 MB/sec execute 337 sec latency 325.112 ms 2024-09-21T01:37:51.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 329671 3.85 MB/sec execute 340 sec latency 53.258 ms 2024-09-21T01:37:51.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1126711 13.48 MB/sec execute 338 sec latency 275.163 ms 2024-09-21T01:37:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:51 smithi090 ceph-mon[35142]: pgmap v502: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.7 MiB/s wr, 95 op/s 2024-09-21T01:37:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:51 smithi045 ceph-mon[28361]: pgmap v502: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.7 MiB/s wr, 95 op/s 2024-09-21T01:37:52.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 330273 3.85 MB/sec execute 341 sec latency 80.985 ms 2024-09-21T01:37:52.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1126772 13.44 MB/sec execute 339 sec latency 308.481 ms 2024-09-21T01:37:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:37:53.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 330776 3.84 MB/sec execute 342 sec latency 102.997 ms 2024-09-21T01:37:53.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1126850 13.40 MB/sec execute 340 sec latency 316.817 ms 2024-09-21T01:37:53.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:53 smithi090 ceph-mon[35142]: pgmap v503: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-21T01:37:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:53 smithi045 ceph-mon[28361]: pgmap v503: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-21T01:37:54.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 331494 3.85 MB/sec execute 343 sec latency 44.486 ms 2024-09-21T01:37:54.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1126893 13.36 MB/sec execute 341 sec latency 283.107 ms 2024-09-21T01:37:55.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 332227 3.84 MB/sec execute 344 sec latency 54.057 ms 2024-09-21T01:37:55.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1127604 13.33 MB/sec execute 342 sec latency 333.425 ms 2024-09-21T01:37:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:55 smithi090 ceph-mon[35142]: pgmap v504: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 76 op/s 2024-09-21T01:37:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:55 smithi045 ceph-mon[28361]: pgmap v504: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 76 op/s 2024-09-21T01:37:56.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 333073 3.85 MB/sec execute 345 sec latency 92.348 ms 2024-09-21T01:37:56.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1129301 13.32 MB/sec execute 343 sec latency 365.909 ms 2024-09-21T01:37:57.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 333683 3.85 MB/sec execute 346 sec latency 10.077 ms 2024-09-21T01:37:57.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1132757 13.34 MB/sec execute 344 sec latency 41.513 ms 2024-09-21T01:37:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:57 smithi090 ceph-mon[35142]: pgmap v505: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:37:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:57 smithi045 ceph-mon[28361]: pgmap v505: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:37:58.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 334150 3.84 MB/sec execute 347 sec latency 16.029 ms 2024-09-21T01:37:58.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1135488 13.33 MB/sec execute 345 sec latency 66.718 ms 2024-09-21T01:37:58.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:37:58.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:37:58.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:58 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:37:59.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:58 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:37:59.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 334900 3.85 MB/sec execute 348 sec latency 14.014 ms 2024-09-21T01:37:59.569 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1138293 13.35 MB/sec execute 346 sec latency 83.886 ms 2024-09-21T01:37:59.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:59 smithi090 ceph-mon[35142]: pgmap v506: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-21T01:37:59.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:37:59 smithi090 ceph-mon[35142]: Adjusting osd_memory_target on smithi090 to 1680M 2024-09-21T01:38:00.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:59 smithi045 ceph-mon[28361]: pgmap v506: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-21T01:38:00.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:37:59 smithi045 ceph-mon[28361]: Adjusting osd_memory_target on smithi090 to 1680M 2024-09-21T01:38:00.507 INFO:tasks.workunit.client.1.smithi090.stdout: 1 335722 3.84 MB/sec execute 349 sec latency 15.847 ms 2024-09-21T01:38:00.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1141088 13.35 MB/sec execute 347 sec latency 43.784 ms 2024-09-21T01:38:01.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 336691 3.85 MB/sec execute 350 sec latency 11.508 ms 2024-09-21T01:38:01.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1144488 13.37 MB/sec execute 348 sec latency 47.108 ms 2024-09-21T01:38:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:01 smithi045 ceph-mon[28361]: pgmap v507: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.6 MiB/s wr, 90 op/s 2024-09-21T01:38:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:01 smithi090 ceph-mon[35142]: pgmap v507: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.6 MiB/s wr, 90 op/s 2024-09-21T01:38:02.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 337244 3.85 MB/sec execute 351 sec latency 14.028 ms 2024-09-21T01:38:02.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1148025 13.38 MB/sec execute 349 sec latency 33.445 ms 2024-09-21T01:38:03.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 337800 3.84 MB/sec execute 352 sec latency 22.247 ms 2024-09-21T01:38:03.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1151519 13.40 MB/sec execute 350 sec latency 33.410 ms 2024-09-21T01:38:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:03 smithi045 ceph-mon[28361]: pgmap v508: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:38:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:03 smithi090 ceph-mon[35142]: pgmap v508: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:38:04.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 338538 3.85 MB/sec execute 353 sec latency 18.056 ms 2024-09-21T01:38:04.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1154162 13.40 MB/sec execute 351 sec latency 83.220 ms 2024-09-21T01:38:05.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 339345 3.84 MB/sec execute 354 sec latency 14.014 ms 2024-09-21T01:38:05.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1157381 13.41 MB/sec execute 352 sec latency 41.727 ms 2024-09-21T01:38:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:05 smithi045 ceph-mon[28361]: pgmap v509: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:38:06.207 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:05 smithi090 ceph-mon[35142]: pgmap v509: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:38:06.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 340264 3.85 MB/sec execute 355 sec latency 17.153 ms 2024-09-21T01:38:06.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1159820 13.41 MB/sec execute 353 sec latency 113.815 ms 2024-09-21T01:38:07.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 340815 3.85 MB/sec execute 356 sec latency 13.220 ms 2024-09-21T01:38:07.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1162287 13.41 MB/sec execute 354 sec latency 50.004 ms 2024-09-21T01:38:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:07 smithi045 ceph-mon[28361]: pgmap v510: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-21T01:38:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:07 smithi090 ceph-mon[35142]: pgmap v510: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-21T01:38:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:08.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 341352 3.84 MB/sec execute 357 sec latency 22.979 ms 2024-09-21T01:38:08.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1165820 13.43 MB/sec execute 355 sec latency 41.789 ms 2024-09-21T01:38:09.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 342113 3.85 MB/sec execute 358 sec latency 13.557 ms 2024-09-21T01:38:09.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1169251 13.44 MB/sec execute 356 sec latency 96.893 ms 2024-09-21T01:38:10.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:09 smithi045 ceph-mon[28361]: pgmap v511: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:38:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:09 smithi090 ceph-mon[35142]: pgmap v511: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-21T01:38:10.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 342891 3.84 MB/sec execute 359 sec latency 15.618 ms 2024-09-21T01:38:10.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1171868 13.44 MB/sec execute 357 sec latency 55.505 ms 2024-09-21T01:38:11.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 343812 3.85 MB/sec execute 360 sec latency 14.243 ms 2024-09-21T01:38:11.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1174456 13.44 MB/sec execute 358 sec latency 34.093 ms 2024-09-21T01:38:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:11 smithi045 ceph-mon[28361]: pgmap v512: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-09-21T01:38:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:11 smithi090 ceph-mon[35142]: pgmap v512: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-09-21T01:38:12.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 344376 3.85 MB/sec execute 361 sec latency 12.324 ms 2024-09-21T01:38:12.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1176581 13.44 MB/sec execute 359 sec latency 34.988 ms 2024-09-21T01:38:13.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 344939 3.84 MB/sec execute 362 sec latency 22.675 ms 2024-09-21T01:38:13.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1180001 13.45 MB/sec execute 360 sec latency 33.441 ms 2024-09-21T01:38:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:13 smithi045 ceph-mon[28361]: pgmap v513: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-21T01:38:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:13 smithi090 ceph-mon[35142]: pgmap v513: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-21T01:38:14.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 345683 3.85 MB/sec execute 363 sec latency 17.145 ms 2024-09-21T01:38:14.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1183431 13.47 MB/sec execute 361 sec latency 100.099 ms 2024-09-21T01:38:15.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 346451 3.84 MB/sec execute 364 sec latency 22.614 ms 2024-09-21T01:38:15.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1185943 13.47 MB/sec execute 362 sec latency 41.636 ms 2024-09-21T01:38:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:15 smithi045 ceph-mon[28361]: pgmap v514: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-09-21T01:38:16.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:15 smithi090 ceph-mon[35142]: pgmap v514: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 86 op/s 2024-09-21T01:38:16.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 347371 3.85 MB/sec execute 365 sec latency 14.404 ms 2024-09-21T01:38:16.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1188769 13.47 MB/sec execute 363 sec latency 33.431 ms 2024-09-21T01:38:17.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 347975 3.85 MB/sec execute 366 sec latency 12.877 ms 2024-09-21T01:38:17.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1192186 13.48 MB/sec execute 364 sec latency 33.398 ms 2024-09-21T01:38:18.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:17 smithi045 ceph-mon[28361]: pgmap v515: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:38:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:17 smithi090 ceph-mon[35142]: pgmap v515: 129 pgs: 129 active+clean; 147 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:38:18.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 348555 3.84 MB/sec execute 367 sec latency 16.575 ms 2024-09-21T01:38:18.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1194869 13.49 MB/sec execute 365 sec latency 49.725 ms 2024-09-21T01:38:19.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 349316 3.85 MB/sec execute 368 sec latency 16.727 ms 2024-09-21T01:38:19.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1197743 13.49 MB/sec execute 366 sec latency 87.064 ms 2024-09-21T01:38:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:19 smithi045 ceph-mon[28361]: pgmap v516: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:38:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:19 smithi090 ceph-mon[35142]: pgmap v516: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:38:20.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 350082 3.84 MB/sec execute 369 sec latency 13.910 ms 2024-09-21T01:38:20.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1201273 13.51 MB/sec execute 367 sec latency 88.247 ms 2024-09-21T01:38:21.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 350932 3.85 MB/sec execute 370 sec latency 13.576 ms 2024-09-21T01:38:21.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1204755 13.52 MB/sec execute 368 sec latency 33.467 ms 2024-09-21T01:38:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:21 smithi045 ceph-mon[28361]: pgmap v517: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-21T01:38:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:21 smithi090 ceph-mon[35142]: pgmap v517: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-21T01:38:22.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 351569 3.85 MB/sec execute 371 sec latency 14.897 ms 2024-09-21T01:38:22.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1206367 13.50 MB/sec execute 369 sec latency 261.570 ms 2024-09-21T01:38:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:23.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 352150 3.84 MB/sec execute 372 sec latency 24.507 ms 2024-09-21T01:38:23.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1209203 13.51 MB/sec execute 370 sec latency 42.229 ms 2024-09-21T01:38:24.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:23 smithi045 ceph-mon[28361]: pgmap v518: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:38:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:23 smithi090 ceph-mon[35142]: pgmap v518: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:38:24.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 352945 3.85 MB/sec execute 373 sec latency 13.208 ms 2024-09-21T01:38:24.570 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1212036 13.52 MB/sec execute 371 sec latency 42.063 ms 2024-09-21T01:38:25.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 353707 3.84 MB/sec execute 374 sec latency 19.562 ms 2024-09-21T01:38:25.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1215462 13.53 MB/sec execute 372 sec latency 108.324 ms 2024-09-21T01:38:26.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:25 smithi045 ceph-mon[28361]: pgmap v519: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:38:26.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:25 smithi090 ceph-mon[35142]: pgmap v519: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-21T01:38:26.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 354488 3.85 MB/sec execute 375 sec latency 13.895 ms 2024-09-21T01:38:26.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1218889 13.55 MB/sec execute 373 sec latency 116.661 ms 2024-09-21T01:38:27.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 355158 3.85 MB/sec execute 376 sec latency 10.593 ms 2024-09-21T01:38:27.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1220556 13.53 MB/sec execute 374 sec latency 238.885 ms 2024-09-21T01:38:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:27 smithi045 ceph-mon[28361]: pgmap v520: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:38:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:27 smithi090 ceph-mon[35142]: pgmap v520: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-21T01:38:28.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 355822 3.85 MB/sec execute 377 sec latency 19.780 ms 2024-09-21T01:38:28.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1220976 13.50 MB/sec execute 375 sec latency 298.058 ms 2024-09-21T01:38:29.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 356575 3.85 MB/sec execute 378 sec latency 26.874 ms 2024-09-21T01:38:29.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1221207 13.47 MB/sec execute 376 sec latency 299.033 ms 2024-09-21T01:38:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:29 smithi045 ceph-mon[28361]: pgmap v521: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:38:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:29 smithi090 ceph-mon[35142]: pgmap v521: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:38:30.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 357280 3.84 MB/sec execute 379 sec latency 69.598 ms 2024-09-21T01:38:30.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1221536 13.44 MB/sec execute 377 sec latency 300.056 ms 2024-09-21T01:38:31.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 358027 3.85 MB/sec execute 380 sec latency 52.837 ms 2024-09-21T01:38:31.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1222432 13.42 MB/sec execute 378 sec latency 325.021 ms 2024-09-21T01:38:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:31 smithi045 ceph-mon[28361]: pgmap v522: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T01:38:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:31 smithi090 ceph-mon[35142]: pgmap v522: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T01:38:32.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 358640 3.85 MB/sec execute 381 sec latency 57.737 ms 2024-09-21T01:38:32.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1222442 13.38 MB/sec execute 379 sec latency 355.704 ms 2024-09-21T01:38:33.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 359193 3.84 MB/sec execute 382 sec latency 49.502 ms 2024-09-21T01:38:33.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1222491 13.35 MB/sec execute 380 sec latency 333.453 ms 2024-09-21T01:38:34.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:33 smithi045 ceph-mon[28361]: pgmap v523: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:38:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:33 smithi090 ceph-mon[35142]: pgmap v523: 129 pgs: 129 active+clean; 151 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-21T01:38:34.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 359908 3.84 MB/sec execute 383 sec latency 78.095 ms 2024-09-21T01:38:34.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1222554 13.31 MB/sec execute 381 sec latency 316.916 ms 2024-09-21T01:38:35.508 INFO:tasks.workunit.client.1.smithi090.stdout: 1 360680 3.84 MB/sec execute 384 sec latency 31.985 ms 2024-09-21T01:38:35.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1222561 13.28 MB/sec execute 382 sec latency 266.865 ms 2024-09-21T01:38:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:35 smithi045 ceph-mon[28361]: pgmap v524: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:38:36.208 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:35 smithi090 ceph-mon[35142]: pgmap v524: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-21T01:38:36.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 361551 3.85 MB/sec execute 385 sec latency 39.004 ms 2024-09-21T01:38:36.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1224750 13.27 MB/sec execute 383 sec latency 291.762 ms 2024-09-21T01:38:37.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 362154 3.85 MB/sec execute 386 sec latency 12.126 ms 2024-09-21T01:38:37.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1227719 13.27 MB/sec execute 384 sec latency 50.080 ms 2024-09-21T01:38:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:37 smithi045 ceph-mon[28361]: pgmap v525: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-21T01:38:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:37 smithi090 ceph-mon[35142]: pgmap v525: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-21T01:38:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:38.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 362735 3.84 MB/sec execute 387 sec latency 17.289 ms 2024-09-21T01:38:38.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1230735 13.29 MB/sec execute 385 sec latency 33.361 ms 2024-09-21T01:38:39.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 363508 3.84 MB/sec execute 388 sec latency 13.549 ms 2024-09-21T01:38:39.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1233816 13.30 MB/sec execute 386 sec latency 45.005 ms 2024-09-21T01:38:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:39 smithi045 ceph-mon[28361]: pgmap v526: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:38:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:39 smithi090 ceph-mon[35142]: pgmap v526: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:38:40.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 364246 3.84 MB/sec execute 389 sec latency 17.430 ms 2024-09-21T01:38:40.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1236818 13.30 MB/sec execute 387 sec latency 41.857 ms 2024-09-21T01:38:41.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 365112 3.85 MB/sec execute 390 sec latency 13.997 ms 2024-09-21T01:38:41.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1240290 13.32 MB/sec execute 388 sec latency 26.513 ms 2024-09-21T01:38:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:41 smithi045 ceph-mon[28361]: pgmap v527: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-21T01:38:42.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:41 smithi090 ceph-mon[35142]: pgmap v527: 129 pgs: 129 active+clean; 153 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-21T01:38:42.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 365742 3.85 MB/sec execute 391 sec latency 12.352 ms 2024-09-21T01:38:42.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1243770 13.33 MB/sec execute 389 sec latency 33.415 ms 2024-09-21T01:38:43.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 366383 3.85 MB/sec execute 392 sec latency 17.230 ms 2024-09-21T01:38:43.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1247306 13.35 MB/sec execute 390 sec latency 91.646 ms 2024-09-21T01:38:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:43 smithi045 ceph-mon[28361]: pgmap v528: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:38:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:43 smithi090 ceph-mon[35142]: pgmap v528: 129 pgs: 129 active+clean; 154 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-21T01:38:44.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 367182 3.84 MB/sec execute 393 sec latency 21.097 ms 2024-09-21T01:38:44.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1250798 13.36 MB/sec execute 391 sec latency 33.683 ms 2024-09-21T01:38:45.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 367910 3.84 MB/sec execute 394 sec latency 12.772 ms 2024-09-21T01:38:45.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1253418 13.36 MB/sec execute 392 sec latency 50.056 ms 2024-09-21T01:38:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:45 smithi045 ceph-mon[28361]: pgmap v529: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:38:46.209 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:45 smithi090 ceph-mon[35142]: pgmap v529: 129 pgs: 129 active+clean; 155 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:38:46.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 368672 3.85 MB/sec execute 395 sec latency 15.489 ms 2024-09-21T01:38:46.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1256584 13.37 MB/sec execute 393 sec latency 41.813 ms 2024-09-21T01:38:47.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 369340 3.85 MB/sec execute 396 sec latency 10.013 ms 2024-09-21T01:38:47.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1259685 13.38 MB/sec execute 394 sec latency 33.413 ms 2024-09-21T01:38:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:47 smithi045 ceph-mon[28361]: pgmap v530: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:38:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:47 smithi090 ceph-mon[35142]: pgmap v530: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 101 op/s 2024-09-21T01:38:48.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 370019 3.85 MB/sec execute 397 sec latency 22.855 ms 2024-09-21T01:38:48.571 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1262647 13.38 MB/sec execute 395 sec latency 44.887 ms 2024-09-21T01:38:49.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 370810 3.84 MB/sec execute 398 sec latency 12.846 ms 2024-09-21T01:38:49.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1265395 13.39 MB/sec execute 396 sec latency 33.456 ms 2024-09-21T01:38:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:49 smithi045 ceph-mon[28361]: pgmap v531: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:38:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:49 smithi090 ceph-mon[35142]: pgmap v531: 129 pgs: 129 active+clean; 156 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:38:50.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 371601 3.85 MB/sec execute 399 sec latency 19.689 ms 2024-09-21T01:38:50.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1268590 13.40 MB/sec execute 397 sec latency 141.452 ms 2024-09-21T01:38:51.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 372396 3.85 MB/sec execute 400 sec latency 14.518 ms 2024-09-21T01:38:51.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1272123 13.42 MB/sec execute 398 sec latency 33.457 ms 2024-09-21T01:38:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:51 smithi045 ceph-mon[28361]: pgmap v532: 129 pgs: 129 active+clean; 157 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-21T01:38:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:51 smithi090 ceph-mon[35142]: pgmap v532: 129 pgs: 129 active+clean; 157 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-21T01:38:52.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 372945 3.85 MB/sec execute 401 sec latency 12.852 ms 2024-09-21T01:38:52.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1274715 13.42 MB/sec execute 399 sec latency 116.670 ms 2024-09-21T01:38:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:38:53.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 373628 3.85 MB/sec execute 402 sec latency 17.331 ms 2024-09-21T01:38:53.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1277916 13.43 MB/sec execute 400 sec latency 74.913 ms 2024-09-21T01:38:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:53 smithi045 ceph-mon[28361]: pgmap v533: 129 pgs: 129 active+clean; 157 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:38:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:53 smithi090 ceph-mon[35142]: pgmap v533: 129 pgs: 129 active+clean; 157 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:38:54.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 374449 3.84 MB/sec execute 403 sec latency 13.567 ms 2024-09-21T01:38:54.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1280826 13.43 MB/sec execute 401 sec latency 33.678 ms 2024-09-21T01:38:55.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 375219 3.85 MB/sec execute 404 sec latency 12.732 ms 2024-09-21T01:38:55.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1283017 13.43 MB/sec execute 402 sec latency 93.812 ms 2024-09-21T01:38:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:55 smithi045 ceph-mon[28361]: pgmap v534: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:38:56.209 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:55 smithi090 ceph-mon[35142]: pgmap v534: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:38:56.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 375993 3.85 MB/sec execute 405 sec latency 13.070 ms 2024-09-21T01:38:56.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1286381 13.44 MB/sec execute 403 sec latency 101.828 ms 2024-09-21T01:38:57.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 376509 3.85 MB/sec execute 406 sec latency 20.561 ms 2024-09-21T01:38:57.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1289801 13.45 MB/sec execute 404 sec latency 208.517 ms 2024-09-21T01:38:58.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:57 smithi045 ceph-mon[28361]: pgmap v535: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-21T01:38:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:57 smithi090 ceph-mon[35142]: pgmap v535: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-21T01:38:58.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 377181 3.84 MB/sec execute 407 sec latency 34.113 ms 2024-09-21T01:38:58.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1292422 13.45 MB/sec execute 405 sec latency 66.574 ms 2024-09-21T01:38:59.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 377915 3.84 MB/sec execute 408 sec latency 78.956 ms 2024-09-21T01:38:59.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1295468 13.46 MB/sec execute 406 sec latency 53.488 ms 2024-09-21T01:39:00.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:59 smithi045 ceph-mon[28361]: pgmap v536: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:39:00.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:38:59 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:39:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:59 smithi090 ceph-mon[35142]: pgmap v536: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:39:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:38:59 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:39:00.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 378454 3.84 MB/sec execute 409 sec latency 125.091 ms 2024-09-21T01:39:00.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1297053 13.45 MB/sec execute 407 sec latency 72.710 ms 2024-09-21T01:39:01.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 379309 3.85 MB/sec execute 410 sec latency 133.288 ms 2024-09-21T01:39:01.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1299544 13.45 MB/sec execute 408 sec latency 57.595 ms 2024-09-21T01:39:01.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:01 smithi045 ceph-mon[28361]: pgmap v537: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:39:02.111 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:01 smithi090 ceph-mon[35142]: pgmap v537: 129 pgs: 129 active+clean; 159 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:39:02.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 379866 3.85 MB/sec execute 411 sec latency 12.862 ms 2024-09-21T01:39:02.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1300537 13.43 MB/sec execute 409 sec latency 233.576 ms 2024-09-21T01:39:03.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 380179 3.84 MB/sec execute 412 sec latency 104.917 ms 2024-09-21T01:39:03.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1300595 13.40 MB/sec execute 410 sec latency 404.973 ms 2024-09-21T01:39:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:03 smithi045 ceph-mon[28361]: pgmap v538: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:39:04.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:03 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:04.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:03 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:03 smithi090 ceph-mon[35142]: pgmap v538: 129 pgs: 129 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-21T01:39:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:03 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:03 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:04.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 380645 3.84 MB/sec execute 413 sec latency 97.574 ms 2024-09-21T01:39:04.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1300602 13.36 MB/sec execute 411 sec latency 458.791 ms 2024-09-21T01:39:05.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 381158 3.83 MB/sec execute 414 sec latency 99.843 ms 2024-09-21T01:39:05.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1300679 13.33 MB/sec execute 412 sec latency 358.767 ms 2024-09-21T01:39:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:05 smithi045 ceph-mon[28361]: pgmap v539: 129 pgs: 129 active+clean; 160 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-09-21T01:39:06.210 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:05 smithi090 ceph-mon[35142]: pgmap v539: 129 pgs: 129 active+clean; 160 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-09-21T01:39:06.509 INFO:tasks.workunit.client.1.smithi090.stdout: 1 381765 3.83 MB/sec execute 415 sec latency 106.366 ms 2024-09-21T01:39:06.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1300719 13.30 MB/sec execute 413 sec latency 450.477 ms 2024-09-21T01:39:07.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 382534 3.83 MB/sec execute 416 sec latency 79.320 ms 2024-09-21T01:39:07.572 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1301280 13.28 MB/sec execute 414 sec latency 292.178 ms 2024-09-21T01:39:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:07 smithi045 ceph-mon[28361]: pgmap v540: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:39:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:07 smithi090 ceph-mon[35142]: pgmap v540: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:39:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:08.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 383064 3.84 MB/sec execute 417 sec latency 90.193 ms 2024-09-21T01:39:08.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1301441 13.25 MB/sec execute 415 sec latency 482.765 ms 2024-09-21T01:39:09.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 383514 3.83 MB/sec execute 418 sec latency 121.563 ms 2024-09-21T01:39:09.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1302234 13.22 MB/sec execute 416 sec latency 342.231 ms 2024-09-21T01:39:10.161 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:09 smithi045 ceph-mon[28361]: pgmap v541: 129 pgs: 129 active+clean; 161 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:39:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:09 smithi090 ceph-mon[35142]: pgmap v541: 129 pgs: 129 active+clean; 161 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-21T01:39:10.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 384108 3.82 MB/sec execute 419 sec latency 20.585 ms 2024-09-21T01:39:10.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1304165 13.22 MB/sec execute 417 sec latency 91.880 ms 2024-09-21T01:39:11.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 384905 3.82 MB/sec execute 420 sec latency 18.202 ms 2024-09-21T01:39:11.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1305781 13.20 MB/sec execute 418 sec latency 117.168 ms 2024-09-21T01:39:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:11 smithi045 ceph-mon[28361]: pgmap v542: 129 pgs: 129 active+clean; 162 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.4 MiB/s wr, 79 op/s 2024-09-21T01:39:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:11 smithi090 ceph-mon[35142]: pgmap v542: 129 pgs: 129 active+clean; 162 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.4 MiB/s wr, 79 op/s 2024-09-21T01:39:12.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 385622 3.82 MB/sec execute 421 sec latency 13.969 ms 2024-09-21T01:39:12.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1307717 13.20 MB/sec execute 419 sec latency 57.600 ms 2024-09-21T01:39:13.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 386448 3.83 MB/sec execute 422 sec latency 15.915 ms 2024-09-21T01:39:13.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1311196 13.21 MB/sec execute 420 sec latency 58.386 ms 2024-09-21T01:39:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:13 smithi045 ceph-mon[28361]: pgmap v543: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:39:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:39:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:39:13.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:13.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:13 smithi090 ceph-mon[35142]: pgmap v543: 129 pgs: 129 active+clean; 163 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-21T01:39:13.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:13.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:13.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:39:13.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:39:13.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:39:14.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 387076 3.83 MB/sec execute 423 sec latency 12.978 ms 2024-09-21T01:39:14.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1314686 13.23 MB/sec execute 421 sec latency 33.390 ms 2024-09-21T01:39:15.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 387731 3.83 MB/sec execute 424 sec latency 16.857 ms 2024-09-21T01:39:15.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1317352 13.23 MB/sec execute 422 sec latency 74.892 ms 2024-09-21T01:39:15.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:15 smithi045 ceph-mon[28361]: pgmap v544: 129 pgs: 129 active+clean; 162 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-21T01:39:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:15 smithi090 ceph-mon[35142]: pgmap v544: 129 pgs: 129 active+clean; 162 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-21T01:39:16.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 388508 3.82 MB/sec execute 425 sec latency 14.173 ms 2024-09-21T01:39:16.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1320741 13.24 MB/sec execute 423 sec latency 40.718 ms 2024-09-21T01:39:17.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 389234 3.82 MB/sec execute 426 sec latency 22.360 ms 2024-09-21T01:39:17.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1323567 13.24 MB/sec execute 424 sec latency 33.346 ms 2024-09-21T01:39:17.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:17 smithi090 ceph-mon[35142]: pgmap v545: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-09-21T01:39:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:17 smithi045 ceph-mon[28361]: pgmap v545: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-09-21T01:39:18.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 390008 3.83 MB/sec execute 427 sec latency 12.966 ms 2024-09-21T01:39:18.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1326531 13.25 MB/sec execute 425 sec latency 41.712 ms 2024-09-21T01:39:19.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 390672 3.83 MB/sec execute 428 sec latency 10.632 ms 2024-09-21T01:39:19.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1329817 13.26 MB/sec execute 426 sec latency 41.168 ms 2024-09-21T01:39:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:19 smithi090 ceph-mon[35142]: pgmap v546: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:39:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:19 smithi045 ceph-mon[28361]: pgmap v546: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:39:20.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 391341 3.83 MB/sec execute 429 sec latency 16.800 ms 2024-09-21T01:39:20.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1332652 13.27 MB/sec execute 427 sec latency 50.084 ms 2024-09-21T01:39:21.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 392146 3.83 MB/sec execute 430 sec latency 12.518 ms 2024-09-21T01:39:21.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1336013 13.28 MB/sec execute 428 sec latency 118.235 ms 2024-09-21T01:39:21.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:21 smithi090 ceph-mon[35142]: pgmap v547: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.5 MiB/s wr, 87 op/s 2024-09-21T01:39:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:21 smithi045 ceph-mon[28361]: pgmap v547: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.5 MiB/s wr, 87 op/s 2024-09-21T01:39:22.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 392936 3.83 MB/sec execute 431 sec latency 13.110 ms 2024-09-21T01:39:22.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1339554 13.29 MB/sec execute 429 sec latency 35.504 ms 2024-09-21T01:39:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:23.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 393607 3.83 MB/sec execute 432 sec latency 13.131 ms 2024-09-21T01:39:23.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1343045 13.31 MB/sec execute 430 sec latency 26.884 ms 2024-09-21T01:39:23.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:23 smithi090 ceph-mon[35142]: pgmap v548: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:39:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:23 smithi045 ceph-mon[28361]: pgmap v548: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:39:24.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 394249 3.83 MB/sec execute 433 sec latency 13.368 ms 2024-09-21T01:39:24.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1345665 13.31 MB/sec execute 431 sec latency 115.301 ms 2024-09-21T01:39:25.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 394942 3.83 MB/sec execute 434 sec latency 24.059 ms 2024-09-21T01:39:25.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1348829 13.32 MB/sec execute 432 sec latency 56.549 ms 2024-09-21T01:39:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:25 smithi090 ceph-mon[35142]: pgmap v549: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-09-21T01:39:26.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:25 smithi045 ceph-mon[28361]: pgmap v549: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-09-21T01:39:26.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 395748 3.83 MB/sec execute 435 sec latency 20.289 ms 2024-09-21T01:39:26.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1351094 13.31 MB/sec execute 433 sec latency 110.613 ms 2024-09-21T01:39:27.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 396529 3.83 MB/sec execute 436 sec latency 21.919 ms 2024-09-21T01:39:27.573 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1353994 13.32 MB/sec execute 434 sec latency 33.328 ms 2024-09-21T01:39:27.970 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:27 smithi045 ceph-mon[28361]: pgmap v550: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:39:27.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:27 smithi090 ceph-mon[35142]: pgmap v550: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:39:28.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 397332 3.84 MB/sec execute 437 sec latency 11.121 ms 2024-09-21T01:39:28.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1357427 13.33 MB/sec execute 435 sec latency 48.489 ms 2024-09-21T01:39:29.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 397841 3.83 MB/sec execute 438 sec latency 15.324 ms 2024-09-21T01:39:29.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1360900 13.34 MB/sec execute 436 sec latency 46.225 ms 2024-09-21T01:39:29.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:29 smithi090 ceph-mon[35142]: pgmap v551: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-09-21T01:39:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:29 smithi045 ceph-mon[28361]: pgmap v551: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-09-21T01:39:30.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 398530 3.83 MB/sec execute 439 sec latency 17.028 ms 2024-09-21T01:39:30.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1364313 13.36 MB/sec execute 437 sec latency 216.832 ms 2024-09-21T01:39:31.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 399364 3.83 MB/sec execute 440 sec latency 22.262 ms 2024-09-21T01:39:31.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1366614 13.35 MB/sec execute 438 sec latency 100.018 ms 2024-09-21T01:39:31.791 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:31 smithi090 ceph-mon[35142]: pgmap v552: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-09-21T01:39:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:31 smithi045 ceph-mon[28361]: pgmap v552: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-09-21T01:39:32.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 399932 3.82 MB/sec execute 441 sec latency 67.203 ms 2024-09-21T01:39:32.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1369560 13.35 MB/sec execute 439 sec latency 41.990 ms 2024-09-21T01:39:33.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 400798 3.83 MB/sec execute 442 sec latency 12.813 ms 2024-09-21T01:39:33.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1372618 13.36 MB/sec execute 440 sec latency 25.194 ms 2024-09-21T01:39:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:33 smithi090 ceph-mon[35142]: pgmap v553: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-09-21T01:39:34.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:33 smithi045 ceph-mon[28361]: pgmap v553: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 83 op/s 2024-09-21T01:39:34.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 401358 3.83 MB/sec execute 443 sec latency 15.084 ms 2024-09-21T01:39:34.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1375823 13.41 MB/sec execute 441 sec latency 116.541 ms 2024-09-21T01:39:35.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 402038 3.83 MB/sec execute 444 sec latency 15.400 ms 2024-09-21T01:39:35.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1379324 13.43 MB/sec execute 442 sec latency 108.365 ms 2024-09-21T01:39:35.987 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:35 smithi045 ceph-mon[28361]: pgmap v554: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:39:35.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:35 smithi090 ceph-mon[35142]: pgmap v554: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:39:36.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 402763 3.82 MB/sec execute 445 sec latency 47.283 ms 2024-09-21T01:39:36.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1379331 13.40 MB/sec execute 443 sec latency 308.543 ms 2024-09-21T01:39:37.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 403422 3.82 MB/sec execute 446 sec latency 51.193 ms 2024-09-21T01:39:37.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1379394 13.37 MB/sec execute 444 sec latency 308.583 ms 2024-09-21T01:39:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:37 smithi090 ceph-mon[35142]: pgmap v555: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:39:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:37 smithi045 ceph-mon[28361]: pgmap v555: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-21T01:39:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:38.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 404177 3.83 MB/sec execute 447 sec latency 102.290 ms 2024-09-21T01:39:38.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1379401 13.34 MB/sec execute 445 sec latency 316.813 ms 2024-09-21T01:39:39.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 404770 3.83 MB/sec execute 448 sec latency 53.289 ms 2024-09-21T01:39:39.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1379515 13.31 MB/sec execute 446 sec latency 341.851 ms 2024-09-21T01:39:39.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:39 smithi045 ceph-mon[28361]: pgmap v556: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-21T01:39:39.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:39 smithi090 ceph-mon[35142]: pgmap v556: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-21T01:39:40.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 405264 3.82 MB/sec execute 449 sec latency 60.963 ms 2024-09-21T01:39:40.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1379584 13.28 MB/sec execute 447 sec latency 383.411 ms 2024-09-21T01:39:41.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 405897 3.82 MB/sec execute 450 sec latency 104.492 ms 2024-09-21T01:39:41.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1380231 13.26 MB/sec execute 448 sec latency 308.517 ms 2024-09-21T01:39:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:41 smithi090 ceph-mon[35142]: pgmap v557: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 73 op/s 2024-09-21T01:39:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:41 smithi045 ceph-mon[28361]: pgmap v557: 129 pgs: 129 active+clean; 170 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.5 MiB/s wr, 73 op/s 2024-09-21T01:39:42.510 INFO:tasks.workunit.client.1.smithi090.stdout: 1 406541 3.82 MB/sec execute 451 sec latency 88.926 ms 2024-09-21T01:39:42.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1381069 13.23 MB/sec execute 449 sec latency 333.523 ms 2024-09-21T01:39:43.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 407341 3.82 MB/sec execute 452 sec latency 13.634 ms 2024-09-21T01:39:43.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1384027 13.24 MB/sec execute 450 sec latency 53.022 ms 2024-09-21T01:39:43.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:43 smithi090 ceph-mon[35142]: pgmap v558: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:39:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:43 smithi045 ceph-mon[28361]: pgmap v558: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:39:44.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 408062 3.83 MB/sec execute 453 sec latency 12.900 ms 2024-09-21T01:39:44.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1386604 13.24 MB/sec execute 451 sec latency 99.910 ms 2024-09-21T01:39:45.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 408561 3.82 MB/sec execute 454 sec latency 13.419 ms 2024-09-21T01:39:45.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1390217 13.26 MB/sec execute 452 sec latency 33.488 ms 2024-09-21T01:39:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:45 smithi090 ceph-mon[35142]: pgmap v559: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-09-21T01:39:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:45 smithi045 ceph-mon[28361]: pgmap v559: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 85 op/s 2024-09-21T01:39:46.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 409236 3.82 MB/sec execute 455 sec latency 20.437 ms 2024-09-21T01:39:46.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1394264 13.28 MB/sec execute 453 sec latency 33.394 ms 2024-09-21T01:39:47.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 410043 3.82 MB/sec execute 456 sec latency 17.374 ms 2024-09-21T01:39:47.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1397164 13.28 MB/sec execute 454 sec latency 156.255 ms 2024-09-21T01:39:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:47 smithi090 ceph-mon[35142]: pgmap v560: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T01:39:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:47 smithi045 ceph-mon[28361]: pgmap v560: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-21T01:39:48.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 410920 3.82 MB/sec execute 457 sec latency 14.780 ms 2024-09-21T01:39:48.574 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1397349 13.25 MB/sec execute 455 sec latency 266.878 ms 2024-09-21T01:39:49.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 411513 3.83 MB/sec execute 458 sec latency 52.669 ms 2024-09-21T01:39:49.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1399694 13.25 MB/sec execute 456 sec latency 305.401 ms 2024-09-21T01:39:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:49 smithi090 ceph-mon[35142]: pgmap v561: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:39:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:49 smithi045 ceph-mon[28361]: pgmap v561: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:39:50.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 412049 3.82 MB/sec execute 459 sec latency 12.173 ms 2024-09-21T01:39:50.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1402903 13.26 MB/sec execute 457 sec latency 33.478 ms 2024-09-21T01:39:51.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 412718 3.82 MB/sec execute 460 sec latency 17.782 ms 2024-09-21T01:39:51.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1406215 13.27 MB/sec execute 458 sec latency 28.909 ms 2024-09-21T01:39:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:51 smithi090 ceph-mon[35142]: pgmap v562: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-09-21T01:39:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:51 smithi045 ceph-mon[28361]: pgmap v562: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.4 MiB/s wr, 77 op/s 2024-09-21T01:39:52.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 413546 3.82 MB/sec execute 461 sec latency 17.211 ms 2024-09-21T01:39:52.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1409414 13.28 MB/sec execute 459 sec latency 58.099 ms 2024-09-21T01:39:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:39:53.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 414283 3.82 MB/sec execute 462 sec latency 14.197 ms 2024-09-21T01:39:53.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1412434 13.29 MB/sec execute 460 sec latency 35.125 ms 2024-09-21T01:39:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:53 smithi090 ceph-mon[35142]: pgmap v563: 129 pgs: 129 active+clean; 173 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-21T01:39:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:53 smithi045 ceph-mon[28361]: pgmap v563: 129 pgs: 129 active+clean; 173 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-21T01:39:54.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 415111 3.83 MB/sec execute 463 sec latency 13.505 ms 2024-09-21T01:39:54.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1415726 13.30 MB/sec execute 461 sec latency 41.526 ms 2024-09-21T01:39:55.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 415632 3.82 MB/sec execute 464 sec latency 12.820 ms 2024-09-21T01:39:55.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1418442 13.30 MB/sec execute 462 sec latency 100.162 ms 2024-09-21T01:39:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:55 smithi090 ceph-mon[35142]: pgmap v564: 129 pgs: 129 active+clean; 173 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-09-21T01:39:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:55 smithi045 ceph-mon[28361]: pgmap v564: 129 pgs: 129 active+clean; 173 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-09-21T01:39:56.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 416325 3.82 MB/sec execute 465 sec latency 19.547 ms 2024-09-21T01:39:56.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1421980 13.31 MB/sec execute 463 sec latency 34.557 ms 2024-09-21T01:39:57.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 417132 3.82 MB/sec execute 466 sec latency 20.342 ms 2024-09-21T01:39:57.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1425402 13.32 MB/sec execute 464 sec latency 146.831 ms 2024-09-21T01:39:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:57 smithi090 ceph-mon[35142]: pgmap v565: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:39:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:57 smithi045 ceph-mon[28361]: pgmap v565: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-21T01:39:58.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 418004 3.82 MB/sec execute 467 sec latency 13.581 ms 2024-09-21T01:39:58.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1427701 13.32 MB/sec execute 465 sec latency 66.795 ms 2024-09-21T01:39:59.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 418702 3.83 MB/sec execute 468 sec latency 13.535 ms 2024-09-21T01:39:59.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1430162 13.32 MB/sec execute 466 sec latency 35.384 ms 2024-09-21T01:39:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:39:59 smithi090 ceph-mon[35142]: pgmap v566: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:40:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:39:59 smithi045 ceph-mon[28361]: pgmap v566: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-21T01:40:00.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 419241 3.82 MB/sec execute 469 sec latency 12.979 ms 2024-09-21T01:40:00.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1432739 13.32 MB/sec execute 467 sec latency 34.880 ms 2024-09-21T01:40:00.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:00 smithi090 ceph-mon[35142]: overall HEALTH_OK 2024-09-21T01:40:01.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:00 smithi045 ceph-mon[28361]: overall HEALTH_OK 2024-09-21T01:40:01.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 420008 3.82 MB/sec execute 470 sec latency 12.545 ms 2024-09-21T01:40:01.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1436163 13.33 MB/sec execute 468 sec latency 51.185 ms 2024-09-21T01:40:01.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:01 smithi045 ceph-mon[28361]: pgmap v567: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-21T01:40:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:01 smithi090 ceph-mon[35142]: pgmap v567: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 72 op/s 2024-09-21T01:40:02.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 420806 3.82 MB/sec execute 471 sec latency 20.154 ms 2024-09-21T01:40:02.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1439636 13.35 MB/sec execute 469 sec latency 56.294 ms 2024-09-21T01:40:03.513 INFO:tasks.workunit.client.1.smithi090.stdout: 1 421750 3.83 MB/sec execute 472 sec latency 24.670 ms 2024-09-21T01:40:03.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1443129 13.36 MB/sec execute 470 sec latency 42.283 ms 2024-09-21T01:40:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:03 smithi090 ceph-mon[35142]: pgmap v568: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-09-21T01:40:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:03 smithi045 ceph-mon[28361]: pgmap v568: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-09-21T01:40:04.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 422361 3.83 MB/sec execute 473 sec latency 12.262 ms 2024-09-21T01:40:04.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1444869 13.34 MB/sec execute 471 sec latency 41.718 ms 2024-09-21T01:40:05.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 422887 3.82 MB/sec execute 474 sec latency 15.707 ms 2024-09-21T01:40:05.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1447635 13.35 MB/sec execute 472 sec latency 90.272 ms 2024-09-21T01:40:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:05 smithi090 ceph-mon[35142]: pgmap v569: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-21T01:40:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:05 smithi045 ceph-mon[28361]: pgmap v569: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-21T01:40:06.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 423662 3.82 MB/sec execute 475 sec latency 12.787 ms 2024-09-21T01:40:06.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1451183 13.36 MB/sec execute 473 sec latency 50.067 ms 2024-09-21T01:40:07.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 424468 3.82 MB/sec execute 476 sec latency 14.698 ms 2024-09-21T01:40:07.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1455009 13.38 MB/sec execute 474 sec latency 33.282 ms 2024-09-21T01:40:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:07 smithi090 ceph-mon[35142]: pgmap v570: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T01:40:07.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:07 smithi045 ceph-mon[28361]: pgmap v570: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.9 MiB/s wr, 105 op/s 2024-09-21T01:40:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:08.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 425411 3.83 MB/sec execute 477 sec latency 20.405 ms 2024-09-21T01:40:08.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1458557 13.39 MB/sec execute 475 sec latency 33.322 ms 2024-09-21T01:40:09.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 425953 3.83 MB/sec execute 478 sec latency 14.137 ms 2024-09-21T01:40:09.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1462102 13.40 MB/sec execute 476 sec latency 26.656 ms 2024-09-21T01:40:09.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:09 smithi090 ceph-mon[35142]: pgmap v571: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:40:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:09 smithi045 ceph-mon[28361]: pgmap v571: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-21T01:40:10.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 426475 3.82 MB/sec execute 479 sec latency 17.666 ms 2024-09-21T01:40:10.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1464560 13.40 MB/sec execute 477 sec latency 220.210 ms 2024-09-21T01:40:11.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 427267 3.82 MB/sec execute 480 sec latency 12.405 ms 2024-09-21T01:40:11.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1467987 13.41 MB/sec execute 478 sec latency 111.559 ms 2024-09-21T01:40:11.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:11 smithi090 ceph-mon[35142]: pgmap v572: 129 pgs: 129 active+clean; 177 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:40:12.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:11 smithi045 ceph-mon[28361]: pgmap v572: 129 pgs: 129 active+clean; 177 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-21T01:40:12.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 428029 3.82 MB/sec execute 481 sec latency 59.905 ms 2024-09-21T01:40:12.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1470697 13.42 MB/sec execute 479 sec latency 81.651 ms 2024-09-21T01:40:13.512 INFO:tasks.workunit.client.1.smithi090.stdout: 1 428959 3.83 MB/sec execute 482 sec latency 14.215 ms 2024-09-21T01:40:13.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1473327 13.41 MB/sec execute 480 sec latency 141.736 ms 2024-09-21T01:40:13.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:13 smithi090 ceph-mon[35142]: pgmap v573: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 105 op/s 2024-09-21T01:40:13.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:40:13.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:40:13.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:40:13.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:13 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:40:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:13 smithi045 ceph-mon[28361]: pgmap v573: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 105 op/s 2024-09-21T01:40:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:40:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:40:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:40:14.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:13 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:40:14.511 INFO:tasks.workunit.client.1.smithi090.stdout: 1 429486 3.83 MB/sec execute 483 sec latency 15.322 ms 2024-09-21T01:40:14.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1476285 13.42 MB/sec execute 481 sec latency 33.667 ms 2024-09-21T01:40:15.512 INFO:tasks.workunit.client.1.smithi090.stdout: 1 430028 3.82 MB/sec execute 484 sec latency 17.874 ms 2024-09-21T01:40:15.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1478751 13.42 MB/sec execute 482 sec latency 191.947 ms 2024-09-21T01:40:15.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:15 smithi090 ceph-mon[35142]: pgmap v574: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:40:16.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:15 smithi045 ceph-mon[28361]: pgmap v574: 129 pgs: 129 active+clean; 178 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:40:16.513 INFO:tasks.workunit.client.1.smithi090.stdout: 1 430658 3.82 MB/sec execute 485 sec latency 111.473 ms 2024-09-21T01:40:16.575 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1478828 13.40 MB/sec execute 483 sec latency 308.492 ms 2024-09-21T01:40:17.513 INFO:tasks.workunit.client.1.smithi090.stdout: 1 431400 3.82 MB/sec execute 486 sec latency 48.464 ms 2024-09-21T01:40:17.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1478934 13.37 MB/sec execute 484 sec latency 325.164 ms 2024-09-21T01:40:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:17 smithi045 ceph-mon[28361]: pgmap v575: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 105 op/s 2024-09-21T01:40:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:17 smithi090 ceph-mon[35142]: pgmap v575: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.8 MiB/s wr, 105 op/s 2024-09-21T01:40:18.514 INFO:tasks.workunit.client.1.smithi090.stdout: 1 432117 3.82 MB/sec execute 487 sec latency 95.991 ms 2024-09-21T01:40:18.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1479582 13.35 MB/sec execute 485 sec latency 330.676 ms 2024-09-21T01:40:19.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 432778 3.83 MB/sec execute 488 sec latency 38.155 ms 2024-09-21T01:40:19.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1480289 13.33 MB/sec execute 486 sec latency 300.201 ms 2024-09-21T01:40:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:19 smithi045 ceph-mon[28361]: pgmap v576: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:40:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:19 smithi090 ceph-mon[35142]: pgmap v576: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-21T01:40:20.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 433255 3.82 MB/sec execute 489 sec latency 93.819 ms 2024-09-21T01:40:20.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1480421 13.30 MB/sec execute 487 sec latency 308.302 ms 2024-09-21T01:40:21.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 433940 3.82 MB/sec execute 490 sec latency 17.537 ms 2024-09-21T01:40:21.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1480939 13.28 MB/sec execute 488 sec latency 291.333 ms 2024-09-21T01:40:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:21 smithi045 ceph-mon[28361]: pgmap v577: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:40:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:21 smithi090 ceph-mon[35142]: pgmap v577: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:40:22.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 434639 3.82 MB/sec execute 491 sec latency 98.828 ms 2024-09-21T01:40:22.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1481243 13.26 MB/sec execute 489 sec latency 312.585 ms 2024-09-21T01:40:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:23.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 435347 3.81 MB/sec execute 492 sec latency 13.442 ms 2024-09-21T01:40:23.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1483122 13.25 MB/sec execute 490 sec latency 283.419 ms 2024-09-21T01:40:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:23 smithi045 ceph-mon[28361]: pgmap v578: 129 pgs: 129 active+clean; 180 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-21T01:40:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:23 smithi090 ceph-mon[35142]: pgmap v578: 129 pgs: 129 active+clean; 180 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-21T01:40:24.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 436097 3.82 MB/sec execute 493 sec latency 13.493 ms 2024-09-21T01:40:24.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1486022 13.26 MB/sec execute 491 sec latency 33.381 ms 2024-09-21T01:40:25.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 436750 3.82 MB/sec execute 494 sec latency 14.491 ms 2024-09-21T01:40:25.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1489388 13.27 MB/sec execute 492 sec latency 42.704 ms 2024-09-21T01:40:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:25 smithi045 ceph-mon[28361]: pgmap v579: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-21T01:40:26.212 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:25 smithi090 ceph-mon[35142]: pgmap v579: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-21T01:40:26.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 437399 3.82 MB/sec execute 495 sec latency 16.346 ms 2024-09-21T01:40:26.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1492866 13.28 MB/sec execute 493 sec latency 48.997 ms 2024-09-21T01:40:27.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 438188 3.82 MB/sec execute 496 sec latency 16.240 ms 2024-09-21T01:40:27.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1496354 13.29 MB/sec execute 494 sec latency 42.145 ms 2024-09-21T01:40:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:27 smithi045 ceph-mon[28361]: pgmap v580: 129 pgs: 129 active+clean; 180 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:40:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:27 smithi090 ceph-mon[35142]: pgmap v580: 129 pgs: 129 active+clean; 180 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 90 op/s 2024-09-21T01:40:28.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 438928 3.81 MB/sec execute 497 sec latency 12.506 ms 2024-09-21T01:40:28.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1497555 13.27 MB/sec execute 495 sec latency 250.329 ms 2024-09-21T01:40:29.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 439632 3.82 MB/sec execute 498 sec latency 96.657 ms 2024-09-21T01:40:29.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1498011 13.25 MB/sec execute 496 sec latency 333.684 ms 2024-09-21T01:40:30.070 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:29 smithi045 ceph-mon[28361]: pgmap v581: 129 pgs: 129 active+clean; 182 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:40:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:29 smithi090 ceph-mon[35142]: pgmap v581: 129 pgs: 129 active+clean; 182 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-21T01:40:30.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 440242 3.82 MB/sec execute 499 sec latency 15.831 ms 2024-09-21T01:40:30.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1500024 13.25 MB/sec execute 497 sec latency 289.465 ms 2024-09-21T01:40:31.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 440845 3.82 MB/sec execute 500 sec latency 23.209 ms 2024-09-21T01:40:31.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1503562 13.26 MB/sec execute 498 sec latency 34.201 ms 2024-09-21T01:40:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:31 smithi045 ceph-mon[28361]: pgmap v582: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-21T01:40:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:31 smithi090 ceph-mon[35142]: pgmap v582: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-21T01:40:32.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 441667 3.82 MB/sec execute 501 sec latency 12.695 ms 2024-09-21T01:40:32.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1507046 13.27 MB/sec execute 499 sec latency 36.433 ms 2024-09-21T01:40:33.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 442361 3.81 MB/sec execute 502 sec latency 19.168 ms 2024-09-21T01:40:33.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1510538 13.28 MB/sec execute 500 sec latency 33.278 ms 2024-09-21T01:40:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:33 smithi045 ceph-mon[28361]: pgmap v583: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:40:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:33 smithi090 ceph-mon[35142]: pgmap v583: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-21T01:40:34.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 443183 3.82 MB/sec execute 503 sec latency 20.476 ms 2024-09-21T01:40:34.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1513189 13.29 MB/sec execute 501 sec latency 49.472 ms 2024-09-21T01:40:35.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 443813 3.82 MB/sec execute 504 sec latency 14.663 ms 2024-09-21T01:40:35.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1514144 13.27 MB/sec execute 502 sec latency 266.911 ms 2024-09-21T01:40:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:35 smithi045 ceph-mon[28361]: pgmap v584: 129 pgs: 129 active+clean; 184 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-09-21T01:40:36.213 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:35 smithi090 ceph-mon[35142]: pgmap v584: 129 pgs: 129 active+clean; 184 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 81 op/s 2024-09-21T01:40:36.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 444445 3.82 MB/sec execute 505 sec latency 20.341 ms 2024-09-21T01:40:36.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1514392 13.24 MB/sec execute 503 sec latency 316.935 ms 2024-09-21T01:40:37.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 445231 3.82 MB/sec execute 506 sec latency 13.602 ms 2024-09-21T01:40:37.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1517936 13.26 MB/sec execute 504 sec latency 33.290 ms 2024-09-21T01:40:37.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:37 smithi045 ceph-mon[28361]: pgmap v585: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:40:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:37 smithi090 ceph-mon[35142]: pgmap v585: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-21T01:40:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:38.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 445962 3.81 MB/sec execute 507 sec latency 19.693 ms 2024-09-21T01:40:38.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1521415 13.27 MB/sec execute 505 sec latency 33.649 ms 2024-09-21T01:40:39.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 446738 3.82 MB/sec execute 508 sec latency 12.512 ms 2024-09-21T01:40:39.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1524840 13.28 MB/sec execute 506 sec latency 41.568 ms 2024-09-21T01:40:40.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:39 smithi045 ceph-mon[28361]: pgmap v586: 129 pgs: 129 active+clean; 185 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:40:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:39 smithi090 ceph-mon[35142]: pgmap v586: 129 pgs: 129 active+clean; 185 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:40:40.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 447398 3.82 MB/sec execute 509 sec latency 12.531 ms 2024-09-21T01:40:40.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1528385 13.29 MB/sec execute 507 sec latency 41.035 ms 2024-09-21T01:40:41.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 448068 3.82 MB/sec execute 510 sec latency 17.105 ms 2024-09-21T01:40:41.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1531857 13.30 MB/sec execute 508 sec latency 116.708 ms 2024-09-21T01:40:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:41 smithi045 ceph-mon[28361]: pgmap v587: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:40:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:41 smithi090 ceph-mon[35142]: pgmap v587: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-21T01:40:42.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 448855 3.82 MB/sec execute 511 sec latency 33.568 ms 2024-09-21T01:40:42.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1531864 13.27 MB/sec execute 509 sec latency 266.841 ms 2024-09-21T01:40:43.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 449542 3.81 MB/sec execute 512 sec latency 51.633 ms 2024-09-21T01:40:43.576 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1533543 13.26 MB/sec execute 510 sec latency 300.153 ms 2024-09-21T01:40:44.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:43 smithi045 ceph-mon[28361]: pgmap v588: 129 pgs: 129 active+clean; 185 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:40:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:43 smithi090 ceph-mon[35142]: pgmap v588: 129 pgs: 129 active+clean; 185 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-21T01:40:44.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 450303 3.82 MB/sec execute 513 sec latency 27.922 ms 2024-09-21T01:40:44.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1535484 13.26 MB/sec execute 511 sec latency 219.533 ms 2024-09-21T01:40:45.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 450967 3.82 MB/sec execute 514 sec latency 9.626 ms 2024-09-21T01:40:45.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1539016 13.27 MB/sec execute 512 sec latency 35.517 ms 2024-09-21T01:40:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:45 smithi045 ceph-mon[28361]: pgmap v589: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-21T01:40:46.213 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:45 smithi090 ceph-mon[35142]: pgmap v589: 129 pgs: 129 active+clean; 186 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-21T01:40:46.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 451617 3.82 MB/sec execute 515 sec latency 20.732 ms 2024-09-21T01:40:46.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1542497 13.28 MB/sec execute 513 sec latency 41.975 ms 2024-09-21T01:40:47.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 452404 3.82 MB/sec execute 516 sec latency 13.832 ms 2024-09-21T01:40:47.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1545089 13.28 MB/sec execute 514 sec latency 40.327 ms 2024-09-21T01:40:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:47 smithi045 ceph-mon[28361]: pgmap v590: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:40:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:47 smithi090 ceph-mon[35142]: pgmap v590: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 95 op/s 2024-09-21T01:40:48.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 453109 3.82 MB/sec execute 517 sec latency 13.592 ms 2024-09-21T01:40:48.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1546228 13.27 MB/sec execute 515 sec latency 283.610 ms 2024-09-21T01:40:49.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 453855 3.82 MB/sec execute 518 sec latency 31.213 ms 2024-09-21T01:40:49.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1546793 13.25 MB/sec execute 516 sec latency 308.275 ms 2024-09-21T01:40:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:49 smithi045 ceph-mon[28361]: pgmap v591: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:40:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:49 smithi090 ceph-mon[35142]: pgmap v591: 129 pgs: 129 active+clean; 187 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:40:50.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 454507 3.82 MB/sec execute 519 sec latency 14.263 ms 2024-09-21T01:40:50.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1549596 13.25 MB/sec execute 517 sec latency 254.217 ms 2024-09-21T01:40:51.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 455159 3.82 MB/sec execute 520 sec latency 17.075 ms 2024-09-21T01:40:51.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1552179 13.26 MB/sec execute 518 sec latency 174.740 ms 2024-09-21T01:40:52.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:51 smithi045 ceph-mon[28361]: pgmap v592: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-09-21T01:40:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:51 smithi090 ceph-mon[35142]: pgmap v592: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-09-21T01:40:52.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 455917 3.82 MB/sec execute 521 sec latency 32.000 ms 2024-09-21T01:40:52.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1554290 13.25 MB/sec execute 519 sec latency 330.594 ms 2024-09-21T01:40:53.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:40:53.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 456638 3.82 MB/sec execute 522 sec latency 16.971 ms 2024-09-21T01:40:53.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1557432 13.26 MB/sec execute 520 sec latency 43.422 ms 2024-09-21T01:40:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:53 smithi045 ceph-mon[28361]: pgmap v593: 129 pgs: 129 active+clean; 189 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-09-21T01:40:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:53 smithi090 ceph-mon[35142]: pgmap v593: 129 pgs: 129 active+clean; 189 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-09-21T01:40:54.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 457373 3.82 MB/sec execute 523 sec latency 19.395 ms 2024-09-21T01:40:54.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1560414 13.26 MB/sec execute 521 sec latency 26.293 ms 2024-09-21T01:40:55.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 458049 3.82 MB/sec execute 524 sec latency 11.988 ms 2024-09-21T01:40:55.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1563840 13.27 MB/sec execute 522 sec latency 34.441 ms 2024-09-21T01:40:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:55 smithi045 ceph-mon[28361]: pgmap v594: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:40:56.213 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:55 smithi090 ceph-mon[35142]: pgmap v594: 129 pgs: 129 active+clean; 188 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:40:56.515 INFO:tasks.workunit.client.1.smithi090.stdout: 1 458641 3.84 MB/sec execute 525 sec latency 70.577 ms 2024-09-21T01:40:56.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1567262 13.28 MB/sec execute 523 sec latency 149.847 ms 2024-09-21T01:40:57.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 459462 3.85 MB/sec execute 526 sec latency 51.497 ms 2024-09-21T01:40:57.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1567391 13.26 MB/sec execute 524 sec latency 241.811 ms 2024-09-21T01:40:58.165 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:57 smithi045 ceph-mon[28361]: pgmap v595: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:40:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:57 smithi090 ceph-mon[35142]: pgmap v595: 129 pgs: 129 active+clean; 190 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-21T01:40:58.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 460213 3.85 MB/sec execute 527 sec latency 35.385 ms 2024-09-21T01:40:58.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1567505 13.23 MB/sec execute 525 sec latency 308.834 ms 2024-09-21T01:40:59.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 460953 3.85 MB/sec execute 528 sec latency 21.746 ms 2024-09-21T01:40:59.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1571012 13.25 MB/sec execute 526 sec latency 158.058 ms 2024-09-21T01:41:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:40:59 smithi045 ceph-mon[28361]: pgmap v596: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.6 MiB/s wr, 78 op/s 2024-09-21T01:41:00.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:40:59 smithi090 ceph-mon[35142]: pgmap v596: 129 pgs: 129 active+clean; 200 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.6 MiB/s wr, 78 op/s 2024-09-21T01:41:00.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 461680 3.85 MB/sec execute 529 sec latency 87.102 ms 2024-09-21T01:41:00.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1573046 13.24 MB/sec execute 527 sec latency 317.046 ms 2024-09-21T01:41:01.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 462341 3.85 MB/sec execute 530 sec latency 12.796 ms 2024-09-21T01:41:01.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1576590 13.25 MB/sec execute 528 sec latency 27.258 ms 2024-09-21T01:41:02.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:01 smithi045 ceph-mon[28361]: pgmap v597: 129 pgs: 129 active+clean; 202 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.9 MiB/s wr, 99 op/s 2024-09-21T01:41:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:01 smithi090 ceph-mon[35142]: pgmap v597: 129 pgs: 129 active+clean; 202 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.9 MiB/s wr, 99 op/s 2024-09-21T01:41:02.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 462976 3.85 MB/sec execute 531 sec latency 17.881 ms 2024-09-21T01:41:02.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1579565 13.25 MB/sec execute 529 sec latency 41.785 ms 2024-09-21T01:41:03.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 463750 3.85 MB/sec execute 532 sec latency 46.598 ms 2024-09-21T01:41:03.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1581569 13.25 MB/sec execute 530 sec latency 269.216 ms 2024-09-21T01:41:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:03 smithi045 ceph-mon[28361]: pgmap v598: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.4 MiB/s wr, 107 op/s 2024-09-21T01:41:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:03 smithi090 ceph-mon[35142]: pgmap v598: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.4 MiB/s wr, 107 op/s 2024-09-21T01:41:04.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 464398 3.85 MB/sec execute 533 sec latency 92.186 ms 2024-09-21T01:41:04.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1581751 13.23 MB/sec execute 531 sec latency 293.384 ms 2024-09-21T01:41:05.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 465225 3.85 MB/sec execute 534 sec latency 17.446 ms 2024-09-21T01:41:05.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1585197 13.24 MB/sec execute 532 sec latency 33.338 ms 2024-09-21T01:41:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:05 smithi045 ceph-mon[28361]: pgmap v599: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.1 MiB/s wr, 81 op/s 2024-09-21T01:41:06.213 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:05 smithi090 ceph-mon[35142]: pgmap v599: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.1 MiB/s wr, 81 op/s 2024-09-21T01:41:06.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 465832 3.85 MB/sec execute 535 sec latency 14.930 ms 2024-09-21T01:41:06.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1588658 13.25 MB/sec execute 533 sec latency 28.305 ms 2024-09-21T01:41:07.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 466449 3.85 MB/sec execute 536 sec latency 17.812 ms 2024-09-21T01:41:07.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1592338 13.26 MB/sec execute 534 sec latency 54.786 ms 2024-09-21T01:41:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:07 smithi045 ceph-mon[28361]: pgmap v600: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.4 MiB/s wr, 109 op/s 2024-09-21T01:41:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:07 smithi090 ceph-mon[35142]: pgmap v600: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.4 MiB/s wr, 109 op/s 2024-09-21T01:41:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:08.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 467278 3.85 MB/sec execute 537 sec latency 12.473 ms 2024-09-21T01:41:08.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1595921 13.27 MB/sec execute 535 sec latency 34.342 ms 2024-09-21T01:41:09.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 468008 3.85 MB/sec execute 538 sec latency 13.387 ms 2024-09-21T01:41:09.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1599469 13.28 MB/sec execute 536 sec latency 34.905 ms 2024-09-21T01:41:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:09 smithi045 ceph-mon[28361]: pgmap v601: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 89 op/s 2024-09-21T01:41:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:09 smithi090 ceph-mon[35142]: pgmap v601: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 89 op/s 2024-09-21T01:41:10.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 468770 3.85 MB/sec execute 539 sec latency 55.462 ms 2024-09-21T01:41:10.577 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1600435 13.27 MB/sec execute 537 sec latency 299.276 ms 2024-09-21T01:41:11.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 469314 3.85 MB/sec execute 540 sec latency 127.133 ms 2024-09-21T01:41:11.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1601465 13.26 MB/sec execute 538 sec latency 291.905 ms 2024-09-21T01:41:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:11 smithi045 ceph-mon[28361]: pgmap v602: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.6 MiB/s wr, 104 op/s 2024-09-21T01:41:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:11 smithi090 ceph-mon[35142]: pgmap v602: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.6 MiB/s wr, 104 op/s 2024-09-21T01:41:12.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 469905 3.85 MB/sec execute 541 sec latency 17.974 ms 2024-09-21T01:41:12.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1604854 13.26 MB/sec execute 539 sec latency 33.719 ms 2024-09-21T01:41:13.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 470688 3.85 MB/sec execute 542 sec latency 14.036 ms 2024-09-21T01:41:13.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1607983 13.27 MB/sec execute 540 sec latency 26.020 ms 2024-09-21T01:41:13.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:13 smithi045 ceph-mon[28361]: pgmap v603: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-21T01:41:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:13 smithi090 ceph-mon[35142]: pgmap v603: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 MiB/s wr, 86 op/s 2024-09-21T01:41:14.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 471440 3.85 MB/sec execute 543 sec latency 19.074 ms 2024-09-21T01:41:14.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1609989 13.27 MB/sec execute 541 sec latency 198.299 ms 2024-09-21T01:41:15.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:41:15.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:41:15.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:41:15.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:14 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:41:15.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:41:15.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:41:15.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:41:15.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:14 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:41:15.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 472286 3.85 MB/sec execute 544 sec latency 93.026 ms 2024-09-21T01:41:15.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1610817 13.25 MB/sec execute 542 sec latency 283.193 ms 2024-09-21T01:41:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:15 smithi045 ceph-mon[28361]: pgmap v604: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:41:16.214 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:15 smithi090 ceph-mon[35142]: pgmap v604: 129 pgs: 129 active+clean; 193 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:41:16.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 472861 3.85 MB/sec execute 545 sec latency 15.819 ms 2024-09-21T01:41:16.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1614366 13.26 MB/sec execute 543 sec latency 40.263 ms 2024-09-21T01:41:17.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 473432 3.85 MB/sec execute 546 sec latency 25.483 ms 2024-09-21T01:41:17.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1618164 13.27 MB/sec execute 544 sec latency 32.382 ms 2024-09-21T01:41:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:17 smithi045 ceph-mon[28361]: pgmap v605: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-09-21T01:41:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:17 smithi090 ceph-mon[35142]: pgmap v605: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 100 op/s 2024-09-21T01:41:18.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 474198 3.85 MB/sec execute 547 sec latency 13.563 ms 2024-09-21T01:41:18.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1622165 13.29 MB/sec execute 545 sec latency 33.427 ms 2024-09-21T01:41:19.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 474928 3.85 MB/sec execute 548 sec latency 94.165 ms 2024-09-21T01:41:19.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1624096 13.29 MB/sec execute 546 sec latency 257.957 ms 2024-09-21T01:41:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:19 smithi045 ceph-mon[28361]: pgmap v606: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-21T01:41:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:19 smithi090 ceph-mon[35142]: pgmap v606: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-21T01:41:20.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 475842 3.85 MB/sec execute 549 sec latency 20.422 ms 2024-09-21T01:41:20.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1627593 13.30 MB/sec execute 547 sec latency 49.504 ms 2024-09-21T01:41:21.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 476427 3.85 MB/sec execute 550 sec latency 14.263 ms 2024-09-21T01:41:21.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1630204 13.30 MB/sec execute 548 sec latency 41.498 ms 2024-09-21T01:41:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:21 smithi045 ceph-mon[28361]: pgmap v607: 129 pgs: 129 active+clean; 196 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1.6 MiB/s wr, 92 op/s 2024-09-21T01:41:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:21 smithi090 ceph-mon[35142]: pgmap v607: 129 pgs: 129 active+clean; 196 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1.6 MiB/s wr, 92 op/s 2024-09-21T01:41:22.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 477025 3.85 MB/sec execute 551 sec latency 25.941 ms 2024-09-21T01:41:22.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1631941 13.29 MB/sec execute 549 sec latency 152.900 ms 2024-09-21T01:41:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:23.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 477781 3.85 MB/sec execute 552 sec latency 13.365 ms 2024-09-21T01:41:23.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1632345 13.27 MB/sec execute 550 sec latency 299.630 ms 2024-09-21T01:41:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:23 smithi045 ceph-mon[28361]: pgmap v608: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:41:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:23 smithi090 ceph-mon[35142]: pgmap v608: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-21T01:41:24.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 478500 3.85 MB/sec execute 553 sec latency 70.999 ms 2024-09-21T01:41:24.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1633451 13.26 MB/sec execute 551 sec latency 316.825 ms 2024-09-21T01:41:25.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 479421 3.85 MB/sec execute 554 sec latency 18.836 ms 2024-09-21T01:41:25.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1636418 13.26 MB/sec execute 552 sec latency 50.185 ms 2024-09-21T01:41:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:25 smithi045 ceph-mon[28361]: pgmap v609: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-09-21T01:41:26.214 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:25 smithi090 ceph-mon[35142]: pgmap v609: 129 pgs: 129 active+clean; 195 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 72 op/s 2024-09-21T01:41:26.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 479974 3.85 MB/sec execute 555 sec latency 55.827 ms 2024-09-21T01:41:26.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1638350 13.26 MB/sec execute 553 sec latency 354.959 ms 2024-09-21T01:41:27.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 480542 3.85 MB/sec execute 556 sec latency 16.541 ms 2024-09-21T01:41:27.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1641825 13.27 MB/sec execute 554 sec latency 358.777 ms 2024-09-21T01:41:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:27 smithi045 ceph-mon[28361]: pgmap v610: 129 pgs: 129 active+clean; 198 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.8 MiB/s wr, 106 op/s 2024-09-21T01:41:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:27 smithi090 ceph-mon[35142]: pgmap v610: 129 pgs: 129 active+clean; 198 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.8 MiB/s wr, 106 op/s 2024-09-21T01:41:28.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 481306 3.85 MB/sec execute 557 sec latency 13.198 ms 2024-09-21T01:41:28.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1645313 13.28 MB/sec execute 555 sec latency 35.665 ms 2024-09-21T01:41:29.516 INFO:tasks.workunit.client.1.smithi090.stdout: 1 482080 3.85 MB/sec execute 558 sec latency 16.532 ms 2024-09-21T01:41:29.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1648862 13.29 MB/sec execute 556 sec latency 33.666 ms 2024-09-21T01:41:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:29 smithi045 ceph-mon[28361]: pgmap v611: 129 pgs: 129 active+clean; 198 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:41:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:29 smithi090 ceph-mon[35142]: pgmap v611: 129 pgs: 129 active+clean; 198 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-21T01:41:30.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 482957 3.85 MB/sec execute 559 sec latency 61.570 ms 2024-09-21T01:41:30.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1648921 13.26 MB/sec execute 557 sec latency 441.870 ms 2024-09-21T01:41:31.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 483495 3.85 MB/sec execute 560 sec latency 53.685 ms 2024-09-21T01:41:31.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1649821 13.25 MB/sec execute 558 sec latency 300.147 ms 2024-09-21T01:41:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:31 smithi045 ceph-mon[28361]: pgmap v612: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:41:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:31 smithi090 ceph-mon[35142]: pgmap v612: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-21T01:41:32.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 484034 3.85 MB/sec execute 561 sec latency 17.092 ms 2024-09-21T01:41:32.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1652718 13.25 MB/sec execute 559 sec latency 33.017 ms 2024-09-21T01:41:33.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 484813 3.85 MB/sec execute 562 sec latency 12.844 ms 2024-09-21T01:41:33.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1656198 13.26 MB/sec execute 560 sec latency 31.421 ms 2024-09-21T01:41:34.098 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:33 smithi045 ceph-mon[28361]: pgmap v613: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-21T01:41:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:33 smithi090 ceph-mon[35142]: pgmap v613: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-21T01:41:34.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 485596 3.84 MB/sec execute 563 sec latency 12.640 ms 2024-09-21T01:41:34.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1659622 13.27 MB/sec execute 561 sec latency 50.135 ms 2024-09-21T01:41:35.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 486507 3.85 MB/sec execute 564 sec latency 17.118 ms 2024-09-21T01:41:35.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1663097 13.28 MB/sec execute 562 sec latency 55.659 ms 2024-09-21T01:41:36.171 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:35 smithi045 ceph-mon[28361]: pgmap v614: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-21T01:41:36.217 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:35 smithi090 ceph-mon[35142]: pgmap v614: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-21T01:41:36.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 487069 3.85 MB/sec execute 565 sec latency 13.122 ms 2024-09-21T01:41:36.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1665701 13.28 MB/sec execute 563 sec latency 33.685 ms 2024-09-21T01:41:37.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 487552 3.85 MB/sec execute 566 sec latency 58.433 ms 2024-09-21T01:41:37.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1667395 13.28 MB/sec execute 564 sec latency 272.714 ms 2024-09-21T01:41:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:37 smithi045 ceph-mon[28361]: pgmap v615: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 112 op/s 2024-09-21T01:41:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:37 smithi090 ceph-mon[35142]: pgmap v615: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 112 op/s 2024-09-21T01:41:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:38.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 488294 3.85 MB/sec execute 567 sec latency 14.183 ms 2024-09-21T01:41:38.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1670256 13.28 MB/sec execute 565 sec latency 352.271 ms 2024-09-21T01:41:39.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 489098 3.84 MB/sec execute 568 sec latency 12.853 ms 2024-09-21T01:41:39.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1673738 13.29 MB/sec execute 566 sec latency 41.319 ms 2024-09-21T01:41:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:39 smithi045 ceph-mon[28361]: pgmap v616: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 84 op/s 2024-09-21T01:41:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:39 smithi090 ceph-mon[35142]: pgmap v616: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 84 op/s 2024-09-21T01:41:40.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 490046 3.85 MB/sec execute 569 sec latency 18.597 ms 2024-09-21T01:41:40.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1676333 13.29 MB/sec execute 567 sec latency 41.761 ms 2024-09-21T01:41:41.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 490613 3.85 MB/sec execute 570 sec latency 14.138 ms 2024-09-21T01:41:41.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1677364 13.28 MB/sec execute 568 sec latency 251.801 ms 2024-09-21T01:41:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:41 smithi045 ceph-mon[28361]: pgmap v617: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:41:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:41 smithi090 ceph-mon[35142]: pgmap v617: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:41:42.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 491199 3.85 MB/sec execute 571 sec latency 18.300 ms 2024-09-21T01:41:42.578 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1680898 13.29 MB/sec execute 569 sec latency 33.427 ms 2024-09-21T01:41:43.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 491976 3.85 MB/sec execute 572 sec latency 12.581 ms 2024-09-21T01:41:43.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1684379 13.30 MB/sec execute 570 sec latency 46.966 ms 2024-09-21T01:41:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:43 smithi045 ceph-mon[28361]: pgmap v618: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-09-21T01:41:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:43 smithi090 ceph-mon[35142]: pgmap v618: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-09-21T01:41:44.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 492741 3.84 MB/sec execute 573 sec latency 13.286 ms 2024-09-21T01:41:44.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1687873 13.31 MB/sec execute 571 sec latency 33.552 ms 2024-09-21T01:41:45.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 493622 3.85 MB/sec execute 574 sec latency 24.296 ms 2024-09-21T01:41:45.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1691415 13.32 MB/sec execute 572 sec latency 41.643 ms 2024-09-21T01:41:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:45 smithi045 ceph-mon[28361]: pgmap v619: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:41:46.215 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:45 smithi090 ceph-mon[35142]: pgmap v619: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-21T01:41:46.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 494232 3.85 MB/sec execute 575 sec latency 13.987 ms 2024-09-21T01:41:46.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1693930 13.32 MB/sec execute 573 sec latency 83.922 ms 2024-09-21T01:41:47.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 494794 3.85 MB/sec execute 576 sec latency 37.937 ms 2024-09-21T01:41:47.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1694071 13.29 MB/sec execute 574 sec latency 282.918 ms 2024-09-21T01:41:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:47 smithi045 ceph-mon[28361]: pgmap v620: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 101 op/s 2024-09-21T01:41:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:47 smithi090 ceph-mon[35142]: pgmap v620: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 101 op/s 2024-09-21T01:41:48.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 495464 3.85 MB/sec execute 577 sec latency 60.301 ms 2024-09-21T01:41:48.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1694962 13.28 MB/sec execute 575 sec latency 316.882 ms 2024-09-21T01:41:49.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 496177 3.84 MB/sec execute 578 sec latency 88.699 ms 2024-09-21T01:41:49.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1695011 13.26 MB/sec execute 576 sec latency 275.222 ms 2024-09-21T01:41:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:49 smithi045 ceph-mon[28361]: pgmap v621: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:41:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:49 smithi090 ceph-mon[35142]: pgmap v621: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-21T01:41:50.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 497116 3.85 MB/sec execute 579 sec latency 50.301 ms 2024-09-21T01:41:50.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1696625 13.25 MB/sec execute 577 sec latency 283.490 ms 2024-09-21T01:41:51.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 497659 3.85 MB/sec execute 580 sec latency 14.063 ms 2024-09-21T01:41:51.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1698710 13.24 MB/sec execute 578 sec latency 119.423 ms 2024-09-21T01:41:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:51 smithi045 ceph-mon[28361]: pgmap v622: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:41:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:51 smithi090 ceph-mon[35142]: pgmap v622: 129 pgs: 129 active+clean; 202 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:41:52.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 498157 3.85 MB/sec execute 581 sec latency 23.925 ms 2024-09-21T01:41:52.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1702170 13.25 MB/sec execute 579 sec latency 35.563 ms 2024-09-21T01:41:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:41:53.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 498933 3.85 MB/sec execute 582 sec latency 13.849 ms 2024-09-21T01:41:53.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1705652 13.26 MB/sec execute 580 sec latency 41.753 ms 2024-09-21T01:41:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:53 smithi045 ceph-mon[28361]: pgmap v623: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-09-21T01:41:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:53 smithi090 ceph-mon[35142]: pgmap v623: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 98 op/s 2024-09-21T01:41:54.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 499756 3.84 MB/sec execute 583 sec latency 13.818 ms 2024-09-21T01:41:54.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1709144 13.27 MB/sec execute 581 sec latency 33.470 ms 2024-09-21T01:41:55.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 500678 3.85 MB/sec execute 584 sec latency 52.128 ms 2024-09-21T01:41:55.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1710350 13.26 MB/sec execute 582 sec latency 374.816 ms 2024-09-21T01:41:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:55 smithi045 ceph-mon[28361]: pgmap v624: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-21T01:41:56.215 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:55 smithi090 ceph-mon[35142]: pgmap v624: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-21T01:41:56.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 501187 3.85 MB/sec execute 585 sec latency 34.549 ms 2024-09-21T01:41:56.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1710837 13.24 MB/sec execute 583 sec latency 407.810 ms 2024-09-21T01:41:57.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 501625 3.85 MB/sec execute 586 sec latency 47.712 ms 2024-09-21T01:41:57.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1712748 13.24 MB/sec execute 584 sec latency 283.246 ms 2024-09-21T01:41:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:57 smithi045 ceph-mon[28361]: pgmap v625: 129 pgs: 129 active+clean; 206 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-21T01:41:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:57 smithi090 ceph-mon[35142]: pgmap v625: 129 pgs: 129 active+clean; 206 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-21T01:41:58.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 502413 3.85 MB/sec execute 587 sec latency 14.895 ms 2024-09-21T01:41:58.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1716234 13.25 MB/sec execute 585 sec latency 66.611 ms 2024-09-21T01:41:59.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 503239 3.84 MB/sec execute 588 sec latency 18.971 ms 2024-09-21T01:41:59.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1718888 13.25 MB/sec execute 586 sec latency 41.566 ms 2024-09-21T01:42:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:41:59 smithi045 ceph-mon[28361]: pgmap v626: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-09-21T01:42:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:41:59 smithi090 ceph-mon[35142]: pgmap v626: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 80 op/s 2024-09-21T01:42:00.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 504127 3.85 MB/sec execute 589 sec latency 15.067 ms 2024-09-21T01:42:00.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1722392 13.26 MB/sec execute 587 sec latency 33.655 ms 2024-09-21T01:42:01.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 504724 3.85 MB/sec execute 590 sec latency 13.442 ms 2024-09-21T01:42:01.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1725408 13.26 MB/sec execute 588 sec latency 68.472 ms 2024-09-21T01:42:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:01 smithi045 ceph-mon[28361]: pgmap v627: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:42:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:01 smithi090 ceph-mon[35142]: pgmap v627: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-21T01:42:02.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 505249 3.85 MB/sec execute 591 sec latency 17.285 ms 2024-09-21T01:42:02.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1727179 13.26 MB/sec execute 589 sec latency 123.034 ms 2024-09-21T01:42:03.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 505897 3.85 MB/sec execute 592 sec latency 84.881 ms 2024-09-21T01:42:03.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1727828 13.24 MB/sec execute 590 sec latency 410.406 ms 2024-09-21T01:42:04.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:03 smithi045 ceph-mon[28361]: pgmap v628: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:42:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:03 smithi090 ceph-mon[35142]: pgmap v628: 129 pgs: 129 active+clean; 205 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 98 op/s 2024-09-21T01:42:04.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 506575 3.84 MB/sec execute 593 sec latency 87.069 ms 2024-09-21T01:42:04.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1729484 13.24 MB/sec execute 591 sec latency 323.824 ms 2024-09-21T01:42:05.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 507418 3.85 MB/sec execute 594 sec latency 14.649 ms 2024-09-21T01:42:05.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1732499 13.24 MB/sec execute 592 sec latency 33.402 ms 2024-09-21T01:42:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:05 smithi045 ceph-mon[28361]: pgmap v629: 129 pgs: 129 active+clean; 206 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-09-21T01:42:06.215 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:05 smithi090 ceph-mon[35142]: pgmap v629: 129 pgs: 129 active+clean; 206 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-09-21T01:42:06.517 INFO:tasks.workunit.client.1.smithi090.stdout: 1 508119 3.85 MB/sec execute 595 sec latency 14.730 ms 2024-09-21T01:42:06.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1735176 13.24 MB/sec execute 593 sec latency 41.274 ms 2024-09-21T01:42:07.518 INFO:tasks.workunit.client.1.smithi090.stdout: 1 508636 3.84 MB/sec execute 596 sec latency 12.629 ms 2024-09-21T01:42:07.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1738461 13.25 MB/sec execute 594 sec latency 27.346 ms 2024-09-21T01:42:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:07 smithi045 ceph-mon[28361]: pgmap v630: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-21T01:42:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:07 smithi090 ceph-mon[35142]: pgmap v630: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-21T01:42:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:08.518 INFO:tasks.workunit.client.1.smithi090.stdout: 1 509320 3.84 MB/sec execute 597 sec latency 17.973 ms 2024-09-21T01:42:08.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1741258 13.25 MB/sec execute 595 sec latency 74.918 ms 2024-09-21T01:42:09.518 INFO:tasks.workunit.client.1.smithi090.stdout: 1 510178 3.84 MB/sec execute 598 sec latency 11.457 ms 2024-09-21T01:42:09.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1744723 13.26 MB/sec execute 596 sec latency 33.192 ms 2024-09-21T01:42:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:09 smithi045 ceph-mon[28361]: pgmap v631: 129 pgs: 129 active+clean; 207 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-21T01:42:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:09 smithi090 ceph-mon[35142]: pgmap v631: 129 pgs: 129 active+clean; 207 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-21T01:42:10.518 INFO:tasks.workunit.client.1.smithi090.stdout: 1 510949 3.85 MB/sec execute 599 sec latency 83.353 ms 2024-09-21T01:42:10.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1746395 13.25 MB/sec execute 597 sec latency 510.853 ms 2024-09-21T01:42:11.518 INFO:tasks.workunit.client.1.smithi090.stdout: 1 cleanup 600 sec 2024-09-21T01:42:11.579 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1748951 13.26 MB/sec execute 598 sec latency 50.020 ms 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: 0 cleanup 600 sec 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: Operation Count AvgLat MaxLat 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: ---------------------------------------- 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: NTCreateX 73375 2.274 127.131 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: Close 53934 0.043 1.760 2024-09-21T01:42:11.830 INFO:tasks.workunit.client.1.smithi090.stdout: Rename 3109 6.576 102.285 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Unlink 14796 2.640 133.283 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Deltree 2 117.834 165.097 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Mkdir 1 0.248 0.248 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Qpathinfo 66552 1.372 88.187 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Qfileinfo 11682 0.066 0.738 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Qfsinfo 12204 0.303 9.495 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Sfileinfo 5970 0.552 3.703 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Find 25749 9.750 125.087 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: WriteX 36643 0.155 96.405 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: ReadX 115245 0.004 0.431 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: LockX 240 0.622 1.251 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: UnlockX 240 0.476 0.803 2024-09-21T01:42:11.831 INFO:tasks.workunit.client.1.smithi090.stdout: Flush 5141 2.655 77.279 2024-09-21T01:42:11.832 INFO:tasks.workunit.client.1.smithi090.stdout: 2024-09-21T01:42:11.832 INFO:tasks.workunit.client.1.smithi090.stdout:Throughput 3.84646 MB/sec 1 clients 1 procs max_latency=133.288 ms 2024-09-21T01:42:11.832 INFO:tasks.workunit.client.1.smithi090.stderr:+ dbench 10 2024-09-21T01:42:11.832 INFO:tasks.workunit.client.1.smithi090.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T01:42:11.833 INFO:tasks.workunit.client.1.smithi090.stdout: 2024-09-21T01:42:11.833 INFO:tasks.workunit.client.1.smithi090.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T01:42:11.833 INFO:tasks.workunit.client.1.smithi090.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-21T01:42:11.932 INFO:tasks.workunit.client.1.smithi090.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-21T01:42:11.932 INFO:tasks.workunit.client.1.smithi090.stdout:releasing clients 2024-09-21T01:42:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:11 smithi045 ceph-mon[28361]: pgmap v632: 129 pgs: 129 active+clean; 207 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-21T01:42:12.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:11 smithi090 ceph-mon[35142]: pgmap v632: 129 pgs: 129 active+clean; 207 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-21T01:42:12.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 1751933 13.26 MB/sec execute 599 sec latency 31.179 ms 2024-09-21T01:42:12.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 301 110.75 MB/sec warmup 1 sec latency 34.296 ms 2024-09-21T01:42:13.580 INFO:tasks.workunit.client.0.smithi045.stdout: 1 cleanup 600 sec 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: 0 cleanup 600 sec 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: Operation Count AvgLat MaxLat 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: ---------------------------------------- 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: NTCreateX 253665 0.011 105.303 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: Close 186335 0.001 0.042 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: Rename 10743 0.024 0.112 2024-09-21T01:42:13.588 INFO:tasks.workunit.client.0.smithi045.stdout: Unlink 51227 0.023 95.857 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Deltree 6 1.788 4.001 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Mkdir 3 0.003 0.003 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Qpathinfo 229916 0.005 1.016 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Qfileinfo 40282 0.001 0.050 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Qfsinfo 42163 0.002 0.037 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Sfileinfo 20668 0.006 1.016 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: Find 88889 0.017 0.143 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: WriteX 126409 0.013 1.243 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: ReadX 397645 0.002 1.071 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: LockX 826 0.003 0.021 2024-09-21T01:42:13.589 INFO:tasks.workunit.client.0.smithi045.stdout: UnlockX 826 0.002 0.015 2024-09-21T01:42:13.590 INFO:tasks.workunit.client.0.smithi045.stdout: Flush 17794 33.073 510.845 2024-09-21T01:42:13.590 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-09-21T01:42:13.590 INFO:tasks.workunit.client.0.smithi045.stdout:Throughput 13.2596 MB/sec 1 clients 1 procs max_latency=510.853 ms 2024-09-21T01:42:13.590 INFO:tasks.workunit.client.0.smithi045.stderr:+ dbench 10 2024-09-21T01:42:13.590 INFO:tasks.workunit.client.0.smithi045.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-21T01:42:13.591 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-09-21T01:42:13.591 INFO:tasks.workunit.client.0.smithi045.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-21T01:42:13.591 INFO:tasks.workunit.client.0.smithi045.stdout:9 of 10 processes prepared for launch 0 sec 2024-09-21T01:42:13.698 INFO:tasks.workunit.client.0.smithi045.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-21T01:42:13.699 INFO:tasks.workunit.client.0.smithi045.stdout:releasing clients 2024-09-21T01:42:13.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 548 95.01 MB/sec warmup 2 sec latency 129.377 ms 2024-09-21T01:42:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:13 smithi045 ceph-mon[28361]: pgmap v633: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-21T01:42:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:13 smithi090 ceph-mon[35142]: pgmap v633: 129 pgs: 129 active+clean; 208 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-21T01:42:14.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 669 211.63 MB/sec warmup 1 sec latency 511.294 ms 2024-09-21T01:42:14.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 728 70.91 MB/sec warmup 3 sec latency 165.126 ms 2024-09-21T01:42:15.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 721 106.47 MB/sec warmup 2 sec latency 284.782 ms 2024-09-21T01:42:15.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 871 54.04 MB/sec warmup 4 sec latency 125.926 ms 2024-09-21T01:42:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:15 smithi045 ceph-mon[28361]: pgmap v634: 129 pgs: 129 active+clean; 217 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.3 MiB/s wr, 75 op/s 2024-09-21T01:42:16.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:42:16.216 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:15 smithi090 ceph-mon[35142]: pgmap v634: 129 pgs: 129 active+clean; 217 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.3 MiB/s wr, 75 op/s 2024-09-21T01:42:16.216 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:42:16.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 728 71.02 MB/sec warmup 3 sec latency 368.498 ms 2024-09-21T01:42:16.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 994 44.33 MB/sec warmup 5 sec latency 139.235 ms 2024-09-21T01:42:17.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 791 53.65 MB/sec warmup 4 sec latency 316.616 ms 2024-09-21T01:42:17.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1102 38.25 MB/sec warmup 6 sec latency 194.528 ms 2024-09-21T01:42:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:17 smithi045 ceph-mon[28361]: pgmap v635: 129 pgs: 129 active+clean; 347 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 MiB/s wr, 143 op/s 2024-09-21T01:42:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:17 smithi090 ceph-mon[35142]: pgmap v635: 129 pgs: 129 active+clean; 347 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 14 MiB/s wr, 143 op/s 2024-09-21T01:42:18.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 798 42.95 MB/sec warmup 5 sec latency 284.702 ms 2024-09-21T01:42:18.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1181 32.89 MB/sec warmup 7 sec latency 364.407 ms 2024-09-21T01:42:19.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 879 36.12 MB/sec warmup 6 sec latency 426.425 ms 2024-09-21T01:42:19.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1301 29.64 MB/sec warmup 8 sec latency 166.672 ms 2024-09-21T01:42:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:19 smithi045 ceph-mon[28361]: pgmap v636: 129 pgs: 129 active+clean; 392 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 18 MiB/s wr, 140 op/s 2024-09-21T01:42:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:19 smithi090 ceph-mon[35142]: pgmap v636: 129 pgs: 129 active+clean; 392 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 18 MiB/s wr, 140 op/s 2024-09-21T01:42:20.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 981 31.29 MB/sec warmup 7 sec latency 365.207 ms 2024-09-21T01:42:20.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1458 28.92 MB/sec warmup 9 sec latency 160.219 ms 2024-09-21T01:42:21.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 1634 32.72 MB/sec warmup 8 sec latency 337.222 ms 2024-09-21T01:42:21.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1604 26.16 MB/sec warmup 10 sec latency 121.058 ms 2024-09-21T01:42:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:21 smithi045 ceph-mon[28361]: pgmap v637: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 19 MiB/s wr, 170 op/s 2024-09-21T01:42:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:21 smithi090 ceph-mon[35142]: pgmap v637: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 19 MiB/s wr, 170 op/s 2024-09-21T01:42:22.690 INFO:tasks.workunit.client.0.smithi045.stdout: 10 2070 29.98 MB/sec warmup 9 sec latency 284.507 ms 2024-09-21T01:42:22.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1791 23.92 MB/sec warmup 11 sec latency 93.888 ms 2024-09-21T01:42:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:23.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 2500 28.98 MB/sec warmup 10 sec latency 386.154 ms 2024-09-21T01:42:23.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 1914 22.02 MB/sec warmup 12 sec latency 162.195 ms 2024-09-21T01:42:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:23 smithi045 ceph-mon[28361]: pgmap v638: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 191 op/s 2024-09-21T01:42:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:23 smithi090 ceph-mon[35142]: pgmap v638: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 191 op/s 2024-09-21T01:42:24.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 2977 30.47 MB/sec warmup 11 sec latency 322.453 ms 2024-09-21T01:42:24.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2094 20.50 MB/sec warmup 13 sec latency 74.128 ms 2024-09-21T01:42:25.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 3308 29.47 MB/sec warmup 12 sec latency 375.157 ms 2024-09-21T01:42:25.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2279 19.56 MB/sec warmup 14 sec latency 103.768 ms 2024-09-21T01:42:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:25 smithi045 ceph-mon[28361]: pgmap v639: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 168 op/s 2024-09-21T01:42:26.216 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:25 smithi090 ceph-mon[35142]: pgmap v639: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 168 op/s 2024-09-21T01:42:26.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 3698 28.72 MB/sec warmup 13 sec latency 379.639 ms 2024-09-21T01:42:26.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2380 18.63 MB/sec warmup 15 sec latency 99.607 ms 2024-09-21T01:42:27.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:42:27.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:27 smithi090 ceph-mon[35142]: pgmap v640: 129 pgs: 129 active+clean; 396 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 18 MiB/s wr, 217 op/s 2024-09-21T01:42:27.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:42:27.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:42:27.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:42:27.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:42:27.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:42:27.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:27 smithi045 ceph-mon[28361]: pgmap v640: 129 pgs: 129 active+clean; 396 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 18 MiB/s wr, 217 op/s 2024-09-21T01:42:27.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:42:27.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:42:27.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:42:27.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:42:27.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 4241 28.78 MB/sec warmup 14 sec latency 446.074 ms 2024-09-21T01:42:27.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2478 17.81 MB/sec warmup 16 sec latency 158.124 ms 2024-09-21T01:42:28.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 4259 26.89 MB/sec warmup 15 sec latency 464.901 ms 2024-09-21T01:42:28.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2587 17.05 MB/sec warmup 17 sec latency 162.857 ms 2024-09-21T01:42:29.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:29 smithi045 ceph-mon[28361]: pgmap v641: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6.7 MiB/s wr, 167 op/s 2024-09-21T01:42:29.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 4294 25.26 MB/sec warmup 16 sec latency 467.159 ms 2024-09-21T01:42:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:29 smithi090 ceph-mon[35142]: pgmap v641: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 6.7 MiB/s wr, 167 op/s 2024-09-21T01:42:29.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2711 17.12 MB/sec warmup 18 sec latency 185.464 ms 2024-09-21T01:42:30.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 4300 23.78 MB/sec warmup 17 sec latency 397.482 ms 2024-09-21T01:42:30.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 2913 17.32 MB/sec warmup 19 sec latency 180.278 ms 2024-09-21T01:42:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:31 smithi090 ceph-mon[35142]: pgmap v642: 129 pgs: 129 active+clean; 388 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-09-21T01:42:31.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:31 smithi045 ceph-mon[28361]: pgmap v642: 129 pgs: 129 active+clean; 388 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-09-21T01:42:31.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 4326 22.49 MB/sec warmup 18 sec latency 468.912 ms 2024-09-21T01:42:31.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3009 16.79 MB/sec warmup 20 sec latency 251.427 ms 2024-09-21T01:42:32.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 4461 21.49 MB/sec warmup 19 sec latency 358.176 ms 2024-09-21T01:42:32.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3275 16.83 MB/sec warmup 21 sec latency 222.794 ms 2024-09-21T01:42:33.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:33 smithi045 ceph-mon[28361]: pgmap v643: 129 pgs: 129 active+clean; 393 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.1 MiB/s wr, 149 op/s 2024-09-21T01:42:33.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 5951 23.59 MB/sec warmup 20 sec latency 118.158 ms 2024-09-21T01:42:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:33 smithi090 ceph-mon[35142]: pgmap v643: 129 pgs: 129 active+clean; 393 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.1 MiB/s wr, 149 op/s 2024-09-21T01:42:33.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3305 16.07 MB/sec warmup 22 sec latency 93.712 ms 2024-09-21T01:42:34.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 7248 26.89 MB/sec warmup 21 sec latency 99.263 ms 2024-09-21T01:42:34.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3325 15.38 MB/sec warmup 23 sec latency 86.747 ms 2024-09-21T01:42:35.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:35 smithi045 ceph-mon[28361]: pgmap v644: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.7 MiB/s wr, 123 op/s 2024-09-21T01:42:35.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 7903 27.14 MB/sec warmup 22 sec latency 101.021 ms 2024-09-21T01:42:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:35 smithi090 ceph-mon[35142]: pgmap v644: 129 pgs: 129 active+clean; 394 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.7 MiB/s wr, 123 op/s 2024-09-21T01:42:35.932 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3393 14.90 MB/sec warmup 24 sec latency 91.089 ms 2024-09-21T01:42:36.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 8352 26.89 MB/sec warmup 23 sec latency 125.288 ms 2024-09-21T01:42:36.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3613 15.42 MB/sec warmup 25 sec latency 112.452 ms 2024-09-21T01:42:37.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:37 smithi045 ceph-mon[28361]: pgmap v645: 129 pgs: 129 active+clean; 401 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.4 MiB/s wr, 132 op/s 2024-09-21T01:42:37.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 9187 27.18 MB/sec warmup 24 sec latency 439.438 ms 2024-09-21T01:42:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:37 smithi090 ceph-mon[35142]: pgmap v645: 129 pgs: 129 active+clean; 401 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.4 MiB/s wr, 132 op/s 2024-09-21T01:42:37.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3745 15.10 MB/sec warmup 26 sec latency 149.902 ms 2024-09-21T01:42:38.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:38.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 9606 26.99 MB/sec warmup 25 sec latency 496.525 ms 2024-09-21T01:42:38.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:38.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3846 14.74 MB/sec warmup 27 sec latency 141.066 ms 2024-09-21T01:42:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 9955 27.24 MB/sec warmup 26 sec latency 416.302 ms 2024-09-21T01:42:39.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3912 14.28 MB/sec warmup 28 sec latency 225.187 ms 2024-09-21T01:42:39.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:39 smithi090 ceph-mon[35142]: pgmap v646: 129 pgs: 129 active+clean; 403 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 98 op/s 2024-09-21T01:42:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:39 smithi045 ceph-mon[28361]: pgmap v646: 129 pgs: 129 active+clean; 403 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 98 op/s 2024-09-21T01:42:40.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 10187 26.77 MB/sec warmup 27 sec latency 503.295 ms 2024-09-21T01:42:40.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 3974 13.82 MB/sec warmup 29 sec latency 211.198 ms 2024-09-21T01:42:41.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 10389 26.21 MB/sec warmup 28 sec latency 404.700 ms 2024-09-21T01:42:41.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4036 13.36 MB/sec warmup 30 sec latency 249.282 ms 2024-09-21T01:42:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:41 smithi090 ceph-mon[35142]: pgmap v647: 129 pgs: 129 active+clean; 404 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.0 MiB/s wr, 83 op/s 2024-09-21T01:42:42.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:41 smithi045 ceph-mon[28361]: pgmap v647: 129 pgs: 129 active+clean; 404 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.0 MiB/s wr, 83 op/s 2024-09-21T01:42:42.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 11149 26.74 MB/sec warmup 29 sec latency 426.976 ms 2024-09-21T01:42:42.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4134 12.94 MB/sec warmup 31 sec latency 186.969 ms 2024-09-21T01:42:43.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 11455 26.28 MB/sec warmup 30 sec latency 91.913 ms 2024-09-21T01:42:43.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4250 12.58 MB/sec warmup 32 sec latency 184.552 ms 2024-09-21T01:42:43.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:43 smithi090 ceph-mon[35142]: pgmap v648: 129 pgs: 129 active+clean; 409 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.2 MiB/s wr, 105 op/s 2024-09-21T01:42:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:43 smithi045 ceph-mon[28361]: pgmap v648: 129 pgs: 129 active+clean; 409 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.2 MiB/s wr, 105 op/s 2024-09-21T01:42:44.691 INFO:tasks.workunit.client.0.smithi045.stdout: 10 12390 26.97 MB/sec warmup 31 sec latency 107.284 ms 2024-09-21T01:42:44.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4406 12.32 MB/sec warmup 33 sec latency 122.889 ms 2024-09-21T01:42:45.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 13852 28.73 MB/sec warmup 32 sec latency 125.325 ms 2024-09-21T01:42:45.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4525 12.07 MB/sec warmup 34 sec latency 119.046 ms 2024-09-21T01:42:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:45 smithi090 ceph-mon[35142]: pgmap v649: 129 pgs: 129 active+clean; 409 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 MiB/s wr, 73 op/s 2024-09-21T01:42:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:45 smithi045 ceph-mon[28361]: pgmap v649: 129 pgs: 129 active+clean; 409 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 MiB/s wr, 73 op/s 2024-09-21T01:42:46.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 14931 29.63 MB/sec warmup 33 sec latency 98.547 ms 2024-09-21T01:42:46.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4655 12.02 MB/sec warmup 35 sec latency 109.596 ms 2024-09-21T01:42:47.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 15103 28.88 MB/sec warmup 34 sec latency 158.633 ms 2024-09-21T01:42:47.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4806 11.80 MB/sec warmup 36 sec latency 92.129 ms 2024-09-21T01:42:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:47 smithi090 ceph-mon[35142]: pgmap v650: 129 pgs: 129 active+clean; 403 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.5 MiB/s wr, 132 op/s 2024-09-21T01:42:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:47 smithi045 ceph-mon[28361]: pgmap v650: 129 pgs: 129 active+clean; 403 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.5 MiB/s wr, 132 op/s 2024-09-21T01:42:48.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 15670 29.08 MB/sec warmup 35 sec latency 408.440 ms 2024-09-21T01:42:48.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 4970 12.20 MB/sec warmup 37 sec latency 158.299 ms 2024-09-21T01:42:49.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 15935 28.54 MB/sec warmup 36 sec latency 617.779 ms 2024-09-21T01:42:49.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5070 11.89 MB/sec warmup 38 sec latency 177.063 ms 2024-09-21T01:42:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:49 smithi090 ceph-mon[35142]: pgmap v651: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.1 MiB/s wr, 139 op/s 2024-09-21T01:42:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:49 smithi045 ceph-mon[28361]: pgmap v651: 129 pgs: 129 active+clean; 392 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.1 MiB/s wr, 139 op/s 2024-09-21T01:42:50.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 16275 27.96 MB/sec warmup 37 sec latency 541.210 ms 2024-09-21T01:42:50.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5166 11.61 MB/sec warmup 39 sec latency 210.310 ms 2024-09-21T01:42:51.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 16625 27.64 MB/sec warmup 38 sec latency 432.047 ms 2024-09-21T01:42:51.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:51 smithi045 ceph-mon[28361]: pgmap v652: 129 pgs: 129 active+clean; 388 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.1 MiB/s wr, 134 op/s 2024-09-21T01:42:51.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5303 11.35 MB/sec warmup 40 sec latency 180.582 ms 2024-09-21T01:42:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:51 smithi090 ceph-mon[35142]: pgmap v652: 129 pgs: 129 active+clean; 388 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.1 MiB/s wr, 134 op/s 2024-09-21T01:42:52.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 16824 27.38 MB/sec warmup 39 sec latency 540.080 ms 2024-09-21T01:42:52.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5414 11.09 MB/sec warmup 41 sec latency 218.036 ms 2024-09-21T01:42:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:42:53.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 17360 27.72 MB/sec warmup 40 sec latency 544.300 ms 2024-09-21T01:42:53.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5559 10.87 MB/sec warmup 42 sec latency 205.276 ms 2024-09-21T01:42:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:53 smithi090 ceph-mon[35142]: pgmap v653: 129 pgs: 129 active+clean; 390 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.8 MiB/s wr, 177 op/s 2024-09-21T01:42:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:53 smithi045 ceph-mon[28361]: pgmap v653: 129 pgs: 129 active+clean; 390 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.8 MiB/s wr, 177 op/s 2024-09-21T01:42:54.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 18461 28.50 MB/sec warmup 41 sec latency 105.098 ms 2024-09-21T01:42:54.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5728 10.68 MB/sec warmup 43 sec latency 96.089 ms 2024-09-21T01:42:55.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 18607 27.91 MB/sec warmup 42 sec latency 180.257 ms 2024-09-21T01:42:55.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5892 10.67 MB/sec warmup 44 sec latency 98.097 ms 2024-09-21T01:42:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:55 smithi090 ceph-mon[35142]: pgmap v654: 129 pgs: 129 active+clean; 390 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 MiB/s wr, 149 op/s 2024-09-21T01:42:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:55 smithi045 ceph-mon[28361]: pgmap v654: 129 pgs: 129 active+clean; 390 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.0 MiB/s wr, 149 op/s 2024-09-21T01:42:56.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 19850 28.52 MB/sec warmup 43 sec latency 109.431 ms 2024-09-21T01:42:56.933 INFO:tasks.workunit.client.1.smithi090.stdout: 10 5991 10.51 MB/sec warmup 45 sec latency 103.101 ms 2024-09-21T01:42:57.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 21363 30.11 MB/sec warmup 44 sec latency 108.282 ms 2024-09-21T01:42:57.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6124 10.50 MB/sec warmup 46 sec latency 123.648 ms 2024-09-21T01:42:57.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:57 smithi090 ceph-mon[35142]: pgmap v655: 129 pgs: 129 active+clean; 392 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-09-21T01:42:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:57 smithi045 ceph-mon[28361]: pgmap v655: 129 pgs: 129 active+clean; 392 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.2 MiB/s wr, 194 op/s 2024-09-21T01:42:58.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 22022 30.22 MB/sec warmup 45 sec latency 393.581 ms 2024-09-21T01:42:58.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6321 10.81 MB/sec warmup 47 sec latency 91.947 ms 2024-09-21T01:42:59.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 22140 29.62 MB/sec warmup 46 sec latency 344.918 ms 2024-09-21T01:42:59.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6602 11.04 MB/sec warmup 48 sec latency 85.494 ms 2024-09-21T01:42:59.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:42:59 smithi090 ceph-mon[35142]: pgmap v656: 129 pgs: 129 active+clean; 393 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:43:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:42:59 smithi045 ceph-mon[28361]: pgmap v656: 129 pgs: 129 active+clean; 393 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 148 op/s 2024-09-21T01:43:00.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 23135 30.04 MB/sec warmup 47 sec latency 103.763 ms 2024-09-21T01:43:00.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6838 11.11 MB/sec warmup 49 sec latency 113.067 ms 2024-09-21T01:43:01.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 24583 31.17 MB/sec warmup 48 sec latency 109.816 ms 2024-09-21T01:43:01.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6863 10.90 MB/sec warmup 50 sec latency 127.774 ms 2024-09-21T01:43:01.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:01 smithi090 ceph-mon[35142]: pgmap v657: 129 pgs: 129 active+clean; 400 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-09-21T01:43:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:01 smithi045 ceph-mon[28361]: pgmap v657: 129 pgs: 129 active+clean; 400 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-09-21T01:43:02.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 25620 31.68 MB/sec warmup 49 sec latency 100.804 ms 2024-09-21T01:43:02.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6888 10.69 MB/sec warmup 51 sec latency 85.516 ms 2024-09-21T01:43:03.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 25875 31.23 MB/sec warmup 50 sec latency 108.353 ms 2024-09-21T01:43:03.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 6980 10.62 MB/sec warmup 52 sec latency 96.916 ms 2024-09-21T01:43:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:03 smithi090 ceph-mon[35142]: pgmap v658: 129 pgs: 129 active+clean; 415 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.7 MiB/s wr, 164 op/s 2024-09-21T01:43:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:03 smithi045 ceph-mon[28361]: pgmap v658: 129 pgs: 129 active+clean; 415 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.7 MiB/s wr, 164 op/s 2024-09-21T01:43:04.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 26965 31.55 MB/sec warmup 51 sec latency 292.499 ms 2024-09-21T01:43:04.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7180 10.90 MB/sec warmup 53 sec latency 172.100 ms 2024-09-21T01:43:05.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 28552 32.93 MB/sec warmup 52 sec latency 101.305 ms 2024-09-21T01:43:05.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7329 10.87 MB/sec warmup 54 sec latency 122.318 ms 2024-09-21T01:43:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:05 smithi090 ceph-mon[35142]: pgmap v659: 129 pgs: 129 active+clean; 416 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.3 MiB/s wr, 117 op/s 2024-09-21T01:43:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:05 smithi045 ceph-mon[28361]: pgmap v659: 129 pgs: 129 active+clean; 416 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.3 MiB/s wr, 117 op/s 2024-09-21T01:43:06.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 29183 32.90 MB/sec warmup 53 sec latency 101.864 ms 2024-09-21T01:43:06.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7424 10.72 MB/sec warmup 55 sec latency 166.248 ms 2024-09-21T01:43:07.692 INFO:tasks.workunit.client.0.smithi045.stdout: 10 29668 32.71 MB/sec warmup 54 sec latency 96.322 ms 2024-09-21T01:43:07.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7514 10.57 MB/sec warmup 56 sec latency 105.427 ms 2024-09-21T01:43:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:07 smithi045 ceph-mon[28361]: pgmap v660: 129 pgs: 129 active+clean; 419 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-09-21T01:43:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:07 smithi090 ceph-mon[35142]: pgmap v660: 129 pgs: 129 active+clean; 419 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.2 MiB/s wr, 148 op/s 2024-09-21T01:43:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:08.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 31059 33.33 MB/sec warmup 55 sec latency 242.061 ms 2024-09-21T01:43:08.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7594 10.38 MB/sec warmup 57 sec latency 128.850 ms 2024-09-21T01:43:09.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 32091 33.99 MB/sec warmup 56 sec latency 308.726 ms 2024-09-21T01:43:09.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7686 10.22 MB/sec warmup 58 sec latency 179.847 ms 2024-09-21T01:43:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:09 smithi045 ceph-mon[28361]: pgmap v661: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 108 op/s 2024-09-21T01:43:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:09 smithi090 ceph-mon[35142]: pgmap v661: 129 pgs: 129 active+clean; 421 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 108 op/s 2024-09-21T01:43:10.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 32730 33.95 MB/sec warmup 57 sec latency 115.716 ms 2024-09-21T01:43:10.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7801 10.07 MB/sec warmup 59 sec latency 140.443 ms 2024-09-21T01:43:11.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 33240 33.75 MB/sec warmup 58 sec latency 112.174 ms 2024-09-21T01:43:11.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 7946 9.96 MB/sec warmup 60 sec latency 116.196 ms 2024-09-21T01:43:12.157 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:11 smithi045 ceph-mon[28361]: pgmap v662: 129 pgs: 129 active+clean; 425 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 119 op/s 2024-09-21T01:43:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:11 smithi090 ceph-mon[35142]: pgmap v662: 129 pgs: 129 active+clean; 425 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.8 MiB/s wr, 119 op/s 2024-09-21T01:43:12.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 35069 34.98 MB/sec warmup 59 sec latency 90.863 ms 2024-09-21T01:43:12.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8071 9.88 MB/sec warmup 61 sec latency 114.482 ms 2024-09-21T01:43:13.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 36215 35.41 MB/sec warmup 60 sec latency 136.774 ms 2024-09-21T01:43:13.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8214 9.87 MB/sec warmup 62 sec latency 95.634 ms 2024-09-21T01:43:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:13 smithi045 ceph-mon[28361]: pgmap v663: 129 pgs: 129 active+clean; 422 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-09-21T01:43:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:13 smithi090 ceph-mon[35142]: pgmap v663: 129 pgs: 129 active+clean; 422 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-09-21T01:43:14.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 36365 34.89 MB/sec warmup 61 sec latency 147.701 ms 2024-09-21T01:43:14.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8356 9.82 MB/sec warmup 63 sec latency 175.033 ms 2024-09-21T01:43:15.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 36435 34.36 MB/sec warmup 62 sec latency 442.676 ms 2024-09-21T01:43:15.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8457 9.81 MB/sec warmup 64 sec latency 201.233 ms 2024-09-21T01:43:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:15 smithi045 ceph-mon[28361]: pgmap v664: 129 pgs: 129 active+clean; 420 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.6 MiB/s wr, 87 op/s 2024-09-21T01:43:16.218 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:15 smithi090 ceph-mon[35142]: pgmap v664: 129 pgs: 129 active+clean; 420 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.6 MiB/s wr, 87 op/s 2024-09-21T01:43:16.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 36757 34.11 MB/sec warmup 63 sec latency 384.690 ms 2024-09-21T01:43:16.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8572 9.89 MB/sec warmup 65 sec latency 215.120 ms 2024-09-21T01:43:17.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 37247 33.99 MB/sec warmup 64 sec latency 459.095 ms 2024-09-21T01:43:17.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8667 9.75 MB/sec warmup 66 sec latency 153.817 ms 2024-09-21T01:43:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:17 smithi045 ceph-mon[28361]: pgmap v665: 129 pgs: 129 active+clean; 407 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.6 MiB/s wr, 167 op/s 2024-09-21T01:43:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:17 smithi090 ceph-mon[35142]: pgmap v665: 129 pgs: 129 active+clean; 407 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.6 MiB/s wr, 167 op/s 2024-09-21T01:43:18.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 37580 33.57 MB/sec warmup 65 sec latency 460.327 ms 2024-09-21T01:43:18.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8799 9.62 MB/sec warmup 67 sec latency 201.451 ms 2024-09-21T01:43:19.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 37613 33.08 MB/sec warmup 66 sec latency 413.309 ms 2024-09-21T01:43:19.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 8942 9.50 MB/sec warmup 68 sec latency 184.643 ms 2024-09-21T01:43:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:19 smithi045 ceph-mon[28361]: pgmap v666: 129 pgs: 129 active+clean; 404 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.1 MiB/s wr, 149 op/s 2024-09-21T01:43:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:19 smithi090 ceph-mon[35142]: pgmap v666: 129 pgs: 129 active+clean; 404 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.1 MiB/s wr, 149 op/s 2024-09-21T01:43:20.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 38060 32.98 MB/sec warmup 67 sec latency 528.069 ms 2024-09-21T01:43:20.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9057 9.38 MB/sec warmup 69 sec latency 160.014 ms 2024-09-21T01:43:21.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 38385 32.93 MB/sec warmup 68 sec latency 544.997 ms 2024-09-21T01:43:21.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9201 9.28 MB/sec warmup 70 sec latency 135.725 ms 2024-09-21T01:43:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:21 smithi045 ceph-mon[28361]: pgmap v667: 129 pgs: 129 active+clean; 404 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-09-21T01:43:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:21 smithi090 ceph-mon[35142]: pgmap v667: 129 pgs: 129 active+clean; 404 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 2.6 MiB/s wr, 151 op/s 2024-09-21T01:43:22.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 38616 32.68 MB/sec warmup 69 sec latency 494.735 ms 2024-09-21T01:43:22.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:22.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9330 9.19 MB/sec warmup 71 sec latency 151.135 ms 2024-09-21T01:43:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:23.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 38814 32.37 MB/sec warmup 70 sec latency 489.118 ms 2024-09-21T01:43:23.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9445 9.18 MB/sec warmup 72 sec latency 176.011 ms 2024-09-21T01:43:24.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:23 smithi045 ceph-mon[28361]: pgmap v668: 129 pgs: 129 active+clean; 406 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-09-21T01:43:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:23 smithi090 ceph-mon[35142]: pgmap v668: 129 pgs: 129 active+clean; 406 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-09-21T01:43:24.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39193 32.20 MB/sec warmup 71 sec latency 460.231 ms 2024-09-21T01:43:24.934 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9518 9.09 MB/sec warmup 73 sec latency 156.851 ms 2024-09-21T01:43:25.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39736 32.16 MB/sec warmup 72 sec latency 461.466 ms 2024-09-21T01:43:25.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9614 9.08 MB/sec warmup 74 sec latency 202.671 ms 2024-09-21T01:43:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:25 smithi045 ceph-mon[28361]: pgmap v669: 129 pgs: 129 active+clean; 406 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-09-21T01:43:26.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:25 smithi090 ceph-mon[35142]: pgmap v669: 129 pgs: 129 active+clean; 406 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.9 MiB/s wr, 166 op/s 2024-09-21T01:43:26.693 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39758 31.73 MB/sec warmup 73 sec latency 417.144 ms 2024-09-21T01:43:26.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9725 9.05 MB/sec warmup 75 sec latency 157.841 ms 2024-09-21T01:43:27.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39777 31.30 MB/sec warmup 74 sec latency 484.071 ms 2024-09-21T01:43:27.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 9842 9.12 MB/sec warmup 76 sec latency 189.524 ms 2024-09-21T01:43:27.971 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:27 smithi045 ceph-mon[28361]: pgmap v670: 129 pgs: 129 active+clean; 402 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-09-21T01:43:27.971 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:27 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:43:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:27 smithi090 ceph-mon[35142]: pgmap v670: 129 pgs: 129 active+clean; 402 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-09-21T01:43:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:27 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:43:28.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39795 30.89 MB/sec warmup 75 sec latency 392.262 ms 2024-09-21T01:43:28.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10039 9.25 MB/sec warmup 77 sec latency 225.003 ms 2024-09-21T01:43:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:43:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:43:29.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:43:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:43:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:43:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:43:29.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39821 30.49 MB/sec warmup 76 sec latency 410.743 ms 2024-09-21T01:43:29.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10185 9.26 MB/sec warmup 78 sec latency 229.329 ms 2024-09-21T01:43:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:29 smithi045 ceph-mon[28361]: pgmap v671: 129 pgs: 129 active+clean; 389 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-09-21T01:43:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:29 smithi090 ceph-mon[35142]: pgmap v671: 129 pgs: 129 active+clean; 389 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.1 MiB/s wr, 126 op/s 2024-09-21T01:43:30.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39860 30.11 MB/sec warmup 77 sec latency 442.000 ms 2024-09-21T01:43:30.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10369 9.30 MB/sec warmup 79 sec latency 209.863 ms 2024-09-21T01:43:31.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 39893 29.73 MB/sec warmup 78 sec latency 424.024 ms 2024-09-21T01:43:31.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10404 9.20 MB/sec warmup 80 sec latency 110.962 ms 2024-09-21T01:43:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:31 smithi045 ceph-mon[28361]: pgmap v672: 129 pgs: 129 active+clean; 395 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-09-21T01:43:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:31 smithi090 ceph-mon[35142]: pgmap v672: 129 pgs: 129 active+clean; 395 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.3 MiB/s wr, 122 op/s 2024-09-21T01:43:32.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 40797 29.95 MB/sec warmup 79 sec latency 351.447 ms 2024-09-21T01:43:32.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10443 9.12 MB/sec warmup 81 sec latency 83.804 ms 2024-09-21T01:43:33.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 42304 30.64 MB/sec warmup 80 sec latency 92.737 ms 2024-09-21T01:43:33.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10532 9.10 MB/sec warmup 82 sec latency 88.579 ms 2024-09-21T01:43:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:33 smithi045 ceph-mon[28361]: pgmap v673: 129 pgs: 129 active+clean; 405 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.2 MiB/s wr, 129 op/s 2024-09-21T01:43:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:33 smithi090 ceph-mon[35142]: pgmap v673: 129 pgs: 129 active+clean; 405 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.2 MiB/s wr, 129 op/s 2024-09-21T01:43:34.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 43334 30.96 MB/sec warmup 81 sec latency 150.707 ms 2024-09-21T01:43:34.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10726 9.27 MB/sec warmup 83 sec latency 126.407 ms 2024-09-21T01:43:35.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 43525 30.64 MB/sec warmup 82 sec latency 92.780 ms 2024-09-21T01:43:35.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10886 9.28 MB/sec warmup 84 sec latency 103.026 ms 2024-09-21T01:43:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:35 smithi045 ceph-mon[28361]: pgmap v674: 129 pgs: 129 active+clean; 407 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.2 MiB/s wr, 83 op/s 2024-09-21T01:43:36.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:35 smithi090 ceph-mon[35142]: pgmap v674: 129 pgs: 129 active+clean; 407 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.2 MiB/s wr, 83 op/s 2024-09-21T01:43:36.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 45139 31.23 MB/sec warmup 83 sec latency 106.131 ms 2024-09-21T01:43:36.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 10976 9.19 MB/sec warmup 85 sec latency 143.881 ms 2024-09-21T01:43:37.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 45719 31.40 MB/sec warmup 84 sec latency 393.123 ms 2024-09-21T01:43:37.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11035 9.10 MB/sec warmup 86 sec latency 209.035 ms 2024-09-21T01:43:38.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:37 smithi045 ceph-mon[28361]: pgmap v675: 129 pgs: 129 active+clean; 410 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 111 op/s 2024-09-21T01:43:38.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:37 smithi090 ceph-mon[35142]: pgmap v675: 129 pgs: 129 active+clean; 410 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.4 KiB/s rd, 4.0 MiB/s wr, 111 op/s 2024-09-21T01:43:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:38.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 46078 31.27 MB/sec warmup 85 sec latency 457.842 ms 2024-09-21T01:43:38.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11088 9.00 MB/sec warmup 87 sec latency 234.708 ms 2024-09-21T01:43:39.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 46292 31.03 MB/sec warmup 86 sec latency 523.437 ms 2024-09-21T01:43:39.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11142 8.90 MB/sec warmup 88 sec latency 414.577 ms 2024-09-21T01:43:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:39 smithi045 ceph-mon[28361]: pgmap v676: 129 pgs: 129 active+clean; 411 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.6 MiB/s wr, 83 op/s 2024-09-21T01:43:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:39 smithi090 ceph-mon[35142]: pgmap v676: 129 pgs: 129 active+clean; 411 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.6 MiB/s wr, 83 op/s 2024-09-21T01:43:40.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 46828 31.01 MB/sec warmup 87 sec latency 635.132 ms 2024-09-21T01:43:40.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11213 8.80 MB/sec warmup 89 sec latency 219.078 ms 2024-09-21T01:43:41.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 46850 30.66 MB/sec warmup 88 sec latency 469.338 ms 2024-09-21T01:43:41.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11289 8.71 MB/sec warmup 90 sec latency 215.075 ms 2024-09-21T01:43:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:41 smithi045 ceph-mon[28361]: pgmap v677: 129 pgs: 129 active+clean; 414 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 88 op/s 2024-09-21T01:43:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:41 smithi090 ceph-mon[35142]: pgmap v677: 129 pgs: 129 active+clean; 414 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.7 MiB/s wr, 88 op/s 2024-09-21T01:43:42.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 46882 30.33 MB/sec warmup 89 sec latency 425.671 ms 2024-09-21T01:43:42.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11371 8.64 MB/sec warmup 91 sec latency 326.576 ms 2024-09-21T01:43:43.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 46888 29.99 MB/sec warmup 90 sec latency 475.718 ms 2024-09-21T01:43:43.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11449 8.56 MB/sec warmup 92 sec latency 218.842 ms 2024-09-21T01:43:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:43 smithi045 ceph-mon[28361]: pgmap v678: 129 pgs: 129 active+clean; 418 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.6 MiB/s wr, 98 op/s 2024-09-21T01:43:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:43 smithi090 ceph-mon[35142]: pgmap v678: 129 pgs: 129 active+clean; 418 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.6 MiB/s wr, 98 op/s 2024-09-21T01:43:44.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 47004 29.69 MB/sec warmup 91 sec latency 450.511 ms 2024-09-21T01:43:44.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11591 8.52 MB/sec warmup 93 sec latency 157.824 ms 2024-09-21T01:43:45.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 48233 29.96 MB/sec warmup 92 sec latency 192.553 ms 2024-09-21T01:43:45.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11734 8.52 MB/sec warmup 94 sec latency 102.244 ms 2024-09-21T01:43:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:45 smithi045 ceph-mon[28361]: pgmap v679: 129 pgs: 129 active+clean; 418 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.7 MiB/s wr, 83 op/s 2024-09-21T01:43:46.219 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:45 smithi090 ceph-mon[35142]: pgmap v679: 129 pgs: 129 active+clean; 418 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.7 MiB/s wr, 83 op/s 2024-09-21T01:43:46.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 49814 30.75 MB/sec warmup 93 sec latency 93.069 ms 2024-09-21T01:43:46.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 11887 8.50 MB/sec warmup 95 sec latency 89.989 ms 2024-09-21T01:43:47.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 50454 30.76 MB/sec warmup 94 sec latency 146.388 ms 2024-09-21T01:43:47.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 12093 8.69 MB/sec warmup 96 sec latency 79.493 ms 2024-09-21T01:43:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:47 smithi045 ceph-mon[28361]: pgmap v680: 129 pgs: 129 active+clean; 403 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 2.9 MiB/s wr, 133 op/s 2024-09-21T01:43:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:47 smithi090 ceph-mon[35142]: pgmap v680: 129 pgs: 129 active+clean; 403 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.4 KiB/s rd, 2.9 MiB/s wr, 133 op/s 2024-09-21T01:43:48.694 INFO:tasks.workunit.client.0.smithi045.stdout: 10 50940 30.67 MB/sec warmup 95 sec latency 99.396 ms 2024-09-21T01:43:48.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 12237 8.61 MB/sec warmup 97 sec latency 85.896 ms 2024-09-21T01:43:49.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 52449 31.11 MB/sec warmup 96 sec latency 143.818 ms 2024-09-21T01:43:49.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 12410 8.53 MB/sec warmup 98 sec latency 83.832 ms 2024-09-21T01:43:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:49 smithi045 ceph-mon[28361]: pgmap v681: 129 pgs: 129 active+clean; 395 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.5 MiB/s wr, 128 op/s 2024-09-21T01:43:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:49 smithi090 ceph-mon[35142]: pgmap v681: 129 pgs: 129 active+clean; 395 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.5 MiB/s wr, 128 op/s 2024-09-21T01:43:50.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 53920 31.75 MB/sec warmup 97 sec latency 150.456 ms 2024-09-21T01:43:50.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 12590 8.47 MB/sec warmup 99 sec latency 87.182 ms 2024-09-21T01:43:51.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 54080 31.47 MB/sec warmup 98 sec latency 94.127 ms 2024-09-21T01:43:51.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 12761 8.41 MB/sec warmup 100 sec latency 110.270 ms 2024-09-21T01:43:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:51 smithi045 ceph-mon[28361]: pgmap v682: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-09-21T01:43:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:51 smithi090 ceph-mon[35142]: pgmap v682: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.6 MiB/s wr, 164 op/s 2024-09-21T01:43:52.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 55339 31.71 MB/sec warmup 99 sec latency 105.635 ms 2024-09-21T01:43:52.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 12934 8.39 MB/sec warmup 101 sec latency 87.174 ms 2024-09-21T01:43:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:43:53.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 56921 32.42 MB/sec warmup 100 sec latency 101.224 ms 2024-09-21T01:43:53.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13065 8.38 MB/sec warmup 102 sec latency 101.434 ms 2024-09-21T01:43:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:53 smithi045 ceph-mon[28361]: pgmap v683: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-09-21T01:43:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:53 smithi090 ceph-mon[35142]: pgmap v683: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.3 MiB/s wr, 172 op/s 2024-09-21T01:43:54.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 57549 32.41 MB/sec warmup 101 sec latency 108.581 ms 2024-09-21T01:43:54.935 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13197 8.39 MB/sec warmup 103 sec latency 96.346 ms 2024-09-21T01:43:55.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 58428 32.56 MB/sec warmup 102 sec latency 110.678 ms 2024-09-21T01:43:55.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13386 8.49 MB/sec warmup 104 sec latency 93.980 ms 2024-09-21T01:43:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:55 smithi045 ceph-mon[28361]: pgmap v684: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.8 MiB/s wr, 156 op/s 2024-09-21T01:43:56.220 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:55 smithi090 ceph-mon[35142]: pgmap v684: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 1.8 MiB/s wr, 156 op/s 2024-09-21T01:43:56.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 59808 32.99 MB/sec warmup 103 sec latency 191.633 ms 2024-09-21T01:43:56.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13655 8.66 MB/sec warmup 105 sec latency 101.170 ms 2024-09-21T01:43:57.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 61036 33.30 MB/sec warmup 104 sec latency 100.500 ms 2024-09-21T01:43:57.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13905 8.72 MB/sec warmup 106 sec latency 93.393 ms 2024-09-21T01:43:58.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:57 smithi045 ceph-mon[28361]: pgmap v685: 129 pgs: 129 active+clean; 405 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-09-21T01:43:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:57 smithi090 ceph-mon[35142]: pgmap v685: 129 pgs: 129 active+clean; 405 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.1 MiB/s wr, 204 op/s 2024-09-21T01:43:58.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 61256 33.03 MB/sec warmup 105 sec latency 116.226 ms 2024-09-21T01:43:58.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13953 8.67 MB/sec warmup 107 sec latency 162.887 ms 2024-09-21T01:43:59.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 62862 33.46 MB/sec warmup 106 sec latency 113.979 ms 2024-09-21T01:43:59.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 13978 8.59 MB/sec warmup 108 sec latency 156.420 ms 2024-09-21T01:44:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:43:59 smithi045 ceph-mon[28361]: pgmap v686: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-09-21T01:44:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:43:59 smithi090 ceph-mon[35142]: pgmap v686: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 166 op/s 2024-09-21T01:44:00.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 63996 33.87 MB/sec warmup 107 sec latency 370.083 ms 2024-09-21T01:44:00.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14059 8.57 MB/sec warmup 109 sec latency 83.184 ms 2024-09-21T01:44:01.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 64635 33.85 MB/sec warmup 108 sec latency 160.840 ms 2024-09-21T01:44:01.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14255 8.73 MB/sec warmup 110 sec latency 116.303 ms 2024-09-21T01:44:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:01 smithi045 ceph-mon[28361]: pgmap v687: 129 pgs: 129 active+clean; 416 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 162 op/s 2024-09-21T01:44:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:01 smithi090 ceph-mon[35142]: pgmap v687: 129 pgs: 129 active+clean; 416 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 162 op/s 2024-09-21T01:44:02.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 65121 33.74 MB/sec warmup 109 sec latency 86.895 ms 2024-09-21T01:44:02.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14403 8.72 MB/sec warmup 111 sec latency 95.899 ms 2024-09-21T01:44:03.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 65972 33.75 MB/sec warmup 110 sec latency 412.691 ms 2024-09-21T01:44:03.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14511 8.68 MB/sec warmup 112 sec latency 108.538 ms 2024-09-21T01:44:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:03 smithi045 ceph-mon[28361]: pgmap v688: 129 pgs: 129 active+clean; 423 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 132 op/s 2024-09-21T01:44:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:03 smithi090 ceph-mon[35142]: pgmap v688: 129 pgs: 129 active+clean; 423 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 132 op/s 2024-09-21T01:44:04.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 67564 34.37 MB/sec warmup 111 sec latency 115.917 ms 2024-09-21T01:44:04.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14605 8.62 MB/sec warmup 113 sec latency 94.609 ms 2024-09-21T01:44:05.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68156 34.34 MB/sec warmup 112 sec latency 335.413 ms 2024-09-21T01:44:05.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14692 8.55 MB/sec warmup 114 sec latency 134.000 ms 2024-09-21T01:44:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:05 smithi045 ceph-mon[28361]: pgmap v689: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-09-21T01:44:06.220 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:05 smithi090 ceph-mon[35142]: pgmap v689: 129 pgs: 129 active+clean; 424 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.6 MiB/s wr, 110 op/s 2024-09-21T01:44:06.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68166 34.04 MB/sec warmup 113 sec latency 448.948 ms 2024-09-21T01:44:06.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14751 8.48 MB/sec warmup 115 sec latency 209.791 ms 2024-09-21T01:44:07.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68189 33.75 MB/sec warmup 114 sec latency 415.080 ms 2024-09-21T01:44:07.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14839 8.41 MB/sec warmup 116 sec latency 212.359 ms 2024-09-21T01:44:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:07 smithi045 ceph-mon[28361]: pgmap v690: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.1 MiB/s wr, 147 op/s 2024-09-21T01:44:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:07 smithi090 ceph-mon[35142]: pgmap v690: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 5.1 MiB/s wr, 147 op/s 2024-09-21T01:44:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:08.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68226 33.46 MB/sec warmup 115 sec latency 458.762 ms 2024-09-21T01:44:08.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 14924 8.35 MB/sec warmup 117 sec latency 230.367 ms 2024-09-21T01:44:09.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68260 33.18 MB/sec warmup 116 sec latency 399.190 ms 2024-09-21T01:44:09.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15037 8.31 MB/sec warmup 118 sec latency 165.095 ms 2024-09-21T01:44:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:09 smithi045 ceph-mon[28361]: pgmap v691: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.1 MiB/s wr, 102 op/s 2024-09-21T01:44:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:09 smithi090 ceph-mon[35142]: pgmap v691: 129 pgs: 129 active+clean; 430 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.1 MiB/s wr, 102 op/s 2024-09-21T01:44:10.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68327 32.91 MB/sec warmup 117 sec latency 415.697 ms 2024-09-21T01:44:10.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15130 8.26 MB/sec warmup 119 sec latency 196.991 ms 2024-09-21T01:44:11.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68571 32.78 MB/sec warmup 118 sec latency 356.959 ms 2024-09-21T01:44:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:11 smithi045 ceph-mon[28361]: pgmap v692: 129 pgs: 129 active+clean; 428 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 3.7 MiB/s wr, 127 op/s 2024-09-21T01:44:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:11 smithi090 ceph-mon[35142]: pgmap v692: 129 pgs: 129 active+clean; 428 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 3.7 MiB/s wr, 127 op/s 2024-09-21T01:44:12.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 68721 32.55 MB/sec warmup 119 sec latency 483.680 ms 2024-09-21T01:44:12.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15353 7.72 MB/sec execute 1 sec latency 181.877 ms 2024-09-21T01:44:13.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15451 5.76 MB/sec execute 2 sec latency 178.827 ms 2024-09-21T01:44:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:13 smithi045 ceph-mon[28361]: pgmap v693: 129 pgs: 129 active+clean; 413 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.6 MiB/s wr, 135 op/s 2024-09-21T01:44:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:13 smithi090 ceph-mon[35142]: pgmap v693: 129 pgs: 129 active+clean; 413 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.6 MiB/s wr, 135 op/s 2024-09-21T01:44:14.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 69506 7.53 MB/sec execute 1 sec latency 589.749 ms 2024-09-21T01:44:14.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15612 9.85 MB/sec execute 3 sec latency 178.963 ms 2024-09-21T01:44:15.695 INFO:tasks.workunit.client.0.smithi045.stdout: 10 69821 9.86 MB/sec execute 2 sec latency 445.759 ms 2024-09-21T01:44:15.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15764 9.04 MB/sec execute 4 sec latency 132.455 ms 2024-09-21T01:44:16.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:15 smithi045 ceph-mon[28361]: pgmap v694: 129 pgs: 129 active+clean; 410 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-09-21T01:44:16.220 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:15 smithi090 ceph-mon[35142]: pgmap v694: 129 pgs: 129 active+clean; 410 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-09-21T01:44:16.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 70098 14.60 MB/sec execute 3 sec latency 423.001 ms 2024-09-21T01:44:16.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15859 7.41 MB/sec execute 5 sec latency 201.854 ms 2024-09-21T01:44:17.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 70449 18.87 MB/sec execute 4 sec latency 421.982 ms 2024-09-21T01:44:17.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 15987 6.35 MB/sec execute 6 sec latency 164.312 ms 2024-09-21T01:44:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:17 smithi045 ceph-mon[28361]: pgmap v695: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.4 MiB/s wr, 169 op/s 2024-09-21T01:44:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:17 smithi090 ceph-mon[35142]: pgmap v695: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.4 MiB/s wr, 169 op/s 2024-09-21T01:44:18.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 70570 16.18 MB/sec execute 5 sec latency 536.886 ms 2024-09-21T01:44:18.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16125 5.64 MB/sec execute 7 sec latency 168.675 ms 2024-09-21T01:44:19.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71087 18.45 MB/sec execute 6 sec latency 466.640 ms 2024-09-21T01:44:19.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16267 5.23 MB/sec execute 8 sec latency 157.788 ms 2024-09-21T01:44:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:19 smithi045 ceph-mon[28361]: pgmap v696: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2024-09-21T01:44:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:19 smithi090 ceph-mon[35142]: pgmap v696: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.1 MiB/s wr, 138 op/s 2024-09-21T01:44:20.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71110 15.84 MB/sec execute 7 sec latency 563.057 ms 2024-09-21T01:44:20.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16371 4.92 MB/sec execute 9 sec latency 179.458 ms 2024-09-21T01:44:21.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71645 17.54 MB/sec execute 8 sec latency 539.056 ms 2024-09-21T01:44:21.936 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16500 5.01 MB/sec execute 10 sec latency 162.512 ms 2024-09-21T01:44:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:21 smithi045 ceph-mon[28361]: pgmap v697: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.0 MiB/s wr, 171 op/s 2024-09-21T01:44:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:21 smithi090 ceph-mon[35142]: pgmap v697: 129 pgs: 129 active+clean; 393 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.0 MiB/s wr, 171 op/s 2024-09-21T01:44:22.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71666 15.65 MB/sec execute 9 sec latency 442.084 ms 2024-09-21T01:44:22.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16579 5.01 MB/sec execute 11 sec latency 185.441 ms 2024-09-21T01:44:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:23.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71693 14.15 MB/sec execute 10 sec latency 442.380 ms 2024-09-21T01:44:23.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16642 4.76 MB/sec execute 12 sec latency 185.959 ms 2024-09-21T01:44:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:23 smithi045 ceph-mon[28361]: pgmap v698: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.8 MiB/s wr, 154 op/s 2024-09-21T01:44:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:23 smithi090 ceph-mon[35142]: pgmap v698: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.8 MiB/s wr, 154 op/s 2024-09-21T01:44:24.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71703 12.88 MB/sec execute 11 sec latency 433.935 ms 2024-09-21T01:44:24.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16735 5.00 MB/sec execute 13 sec latency 186.958 ms 2024-09-21T01:44:25.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 71759 11.92 MB/sec execute 12 sec latency 440.651 ms 2024-09-21T01:44:25.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 16942 6.17 MB/sec execute 14 sec latency 92.810 ms 2024-09-21T01:44:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:25 smithi045 ceph-mon[28361]: pgmap v699: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:44:26.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:25 smithi090 ceph-mon[35142]: pgmap v699: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 126 op/s 2024-09-21T01:44:26.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 72704 14.71 MB/sec execute 13 sec latency 84.297 ms 2024-09-21T01:44:26.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17180 7.22 MB/sec execute 15 sec latency 109.910 ms 2024-09-21T01:44:27.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 74292 20.13 MB/sec execute 14 sec latency 99.400 ms 2024-09-21T01:44:27.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17436 7.74 MB/sec execute 16 sec latency 117.386 ms 2024-09-21T01:44:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:27 smithi045 ceph-mon[28361]: pgmap v700: 129 pgs: 129 active+clean; 396 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-09-21T01:44:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:27 smithi090 ceph-mon[35142]: pgmap v700: 129 pgs: 129 active+clean; 396 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.6 MiB/s wr, 160 op/s 2024-09-21T01:44:28.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 75282 22.23 MB/sec execute 15 sec latency 116.072 ms 2024-09-21T01:44:28.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17496 7.56 MB/sec execute 17 sec latency 97.521 ms 2024-09-21T01:44:29.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:28 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:44:29.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:28 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:44:29.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 75725 22.24 MB/sec execute 16 sec latency 91.011 ms 2024-09-21T01:44:29.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17536 7.32 MB/sec execute 18 sec latency 96.962 ms 2024-09-21T01:44:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:29 smithi045 ceph-mon[28361]: pgmap v701: 129 pgs: 129 active+clean; 404 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-09-21T01:44:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:44:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:44:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:44:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:44:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:44:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:29 smithi090 ceph-mon[35142]: pgmap v701: 129 pgs: 129 active+clean; 404 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.4 KiB/s rd, 2.6 MiB/s wr, 123 op/s 2024-09-21T01:44:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:44:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:44:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:44:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:44:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:29 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:44:30.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 76987 23.90 MB/sec execute 17 sec latency 196.037 ms 2024-09-21T01:44:30.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17610 7.19 MB/sec execute 19 sec latency 104.061 ms 2024-09-21T01:44:31.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 77604 25.48 MB/sec execute 18 sec latency 391.125 ms 2024-09-21T01:44:31.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17767 7.81 MB/sec execute 20 sec latency 144.274 ms 2024-09-21T01:44:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:31 smithi045 ceph-mon[28361]: pgmap v702: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-09-21T01:44:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:31 smithi090 ceph-mon[35142]: pgmap v702: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.8 MiB/s wr, 118 op/s 2024-09-21T01:44:32.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 77764 24.57 MB/sec execute 19 sec latency 547.128 ms 2024-09-21T01:44:32.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 17925 8.02 MB/sec execute 21 sec latency 157.729 ms 2024-09-21T01:44:33.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 77958 23.95 MB/sec execute 20 sec latency 392.784 ms 2024-09-21T01:44:33.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18020 7.87 MB/sec execute 22 sec latency 223.813 ms 2024-09-21T01:44:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:33 smithi045 ceph-mon[28361]: pgmap v703: 129 pgs: 129 active+clean; 415 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.5 MiB/s wr, 111 op/s 2024-09-21T01:44:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:33 smithi090 ceph-mon[35142]: pgmap v703: 129 pgs: 129 active+clean; 415 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.5 MiB/s wr, 111 op/s 2024-09-21T01:44:34.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 78587 24.45 MB/sec execute 21 sec latency 567.735 ms 2024-09-21T01:44:34.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18092 7.58 MB/sec execute 23 sec latency 161.828 ms 2024-09-21T01:44:35.696 INFO:tasks.workunit.client.0.smithi045.stdout: 10 78769 23.79 MB/sec execute 22 sec latency 460.653 ms 2024-09-21T01:44:35.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18156 7.32 MB/sec execute 24 sec latency 255.799 ms 2024-09-21T01:44:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:35 smithi045 ceph-mon[28361]: pgmap v704: 129 pgs: 129 active+clean; 415 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.4 MiB/s wr, 92 op/s 2024-09-21T01:44:36.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:35 smithi090 ceph-mon[35142]: pgmap v704: 129 pgs: 129 active+clean; 415 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.4 MiB/s wr, 92 op/s 2024-09-21T01:44:36.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 78792 22.78 MB/sec execute 23 sec latency 459.107 ms 2024-09-21T01:44:36.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18217 7.04 MB/sec execute 25 sec latency 205.038 ms 2024-09-21T01:44:37.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 78903 21.94 MB/sec execute 24 sec latency 417.385 ms 2024-09-21T01:44:37.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18328 6.79 MB/sec execute 26 sec latency 105.748 ms 2024-09-21T01:44:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:37 smithi045 ceph-mon[28361]: pgmap v705: 129 pgs: 129 active+clean; 421 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.9 MiB/s wr, 127 op/s 2024-09-21T01:44:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:37 smithi090 ceph-mon[35142]: pgmap v705: 129 pgs: 129 active+clean; 421 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.9 MiB/s wr, 127 op/s 2024-09-21T01:44:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:38.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 80160 23.26 MB/sec execute 25 sec latency 107.373 ms 2024-09-21T01:44:38.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18449 6.60 MB/sec execute 27 sec latency 117.411 ms 2024-09-21T01:44:39.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 81382 25.54 MB/sec execute 26 sec latency 114.725 ms 2024-09-21T01:44:39.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18586 6.51 MB/sec execute 28 sec latency 123.742 ms 2024-09-21T01:44:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:39 smithi045 ceph-mon[28361]: pgmap v706: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.0 MiB/s wr, 92 op/s 2024-09-21T01:44:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:39 smithi090 ceph-mon[35142]: pgmap v706: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.0 MiB/s wr, 92 op/s 2024-09-21T01:44:40.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 82374 26.51 MB/sec execute 27 sec latency 136.332 ms 2024-09-21T01:44:40.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18709 6.41 MB/sec execute 29 sec latency 167.761 ms 2024-09-21T01:44:41.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 82509 25.68 MB/sec execute 28 sec latency 178.220 ms 2024-09-21T01:44:41.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 18854 6.52 MB/sec execute 30 sec latency 115.397 ms 2024-09-21T01:44:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:41 smithi045 ceph-mon[28361]: pgmap v707: 129 pgs: 129 active+clean; 420 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.3 MiB/s wr, 92 op/s 2024-09-21T01:44:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:41 smithi090 ceph-mon[35142]: pgmap v707: 129 pgs: 129 active+clean; 420 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.3 MiB/s wr, 92 op/s 2024-09-21T01:44:42.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 83712 26.65 MB/sec execute 29 sec latency 117.134 ms 2024-09-21T01:44:42.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 19012 6.54 MB/sec execute 31 sec latency 92.044 ms 2024-09-21T01:44:43.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 84949 28.55 MB/sec execute 30 sec latency 107.015 ms 2024-09-21T01:44:43.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 19184 6.90 MB/sec execute 32 sec latency 112.609 ms 2024-09-21T01:44:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:43 smithi045 ceph-mon[28361]: pgmap v708: 129 pgs: 129 active+clean; 410 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 125 op/s 2024-09-21T01:44:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:43 smithi090 ceph-mon[35142]: pgmap v708: 129 pgs: 129 active+clean; 410 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.4 MiB/s wr, 125 op/s 2024-09-21T01:44:44.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 85925 29.27 MB/sec execute 31 sec latency 93.410 ms 2024-09-21T01:44:44.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 19349 6.90 MB/sec execute 33 sec latency 93.580 ms 2024-09-21T01:44:45.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 86358 29.04 MB/sec execute 32 sec latency 103.133 ms 2024-09-21T01:44:45.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 19517 6.74 MB/sec execute 34 sec latency 97.709 ms 2024-09-21T01:44:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:45 smithi045 ceph-mon[28361]: pgmap v709: 129 pgs: 129 active+clean; 411 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 139 op/s 2024-09-21T01:44:46.222 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:45 smithi090 ceph-mon[35142]: pgmap v709: 129 pgs: 129 active+clean; 411 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 139 op/s 2024-09-21T01:44:46.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 87115 29.07 MB/sec execute 33 sec latency 374.977 ms 2024-09-21T01:44:46.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 19695 6.60 MB/sec execute 35 sec latency 76.941 ms 2024-09-21T01:44:47.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 88171 30.29 MB/sec execute 34 sec latency 363.401 ms 2024-09-21T01:44:47.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 19869 6.52 MB/sec execute 36 sec latency 103.961 ms 2024-09-21T01:44:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:47 smithi045 ceph-mon[28361]: pgmap v710: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.6 MiB/s wr, 181 op/s 2024-09-21T01:44:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:47 smithi090 ceph-mon[35142]: pgmap v710: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.6 MiB/s wr, 181 op/s 2024-09-21T01:44:48.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 89431 31.36 MB/sec execute 35 sec latency 102.520 ms 2024-09-21T01:44:48.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 20037 6.47 MB/sec execute 37 sec latency 85.317 ms 2024-09-21T01:44:49.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 89566 30.57 MB/sec execute 36 sec latency 100.481 ms 2024-09-21T01:44:49.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 20193 6.53 MB/sec execute 38 sec latency 85.679 ms 2024-09-21T01:44:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:49 smithi045 ceph-mon[28361]: pgmap v711: 129 pgs: 129 active+clean; 402 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 159 op/s 2024-09-21T01:44:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:49 smithi090 ceph-mon[35142]: pgmap v711: 129 pgs: 129 active+clean; 402 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 159 op/s 2024-09-21T01:44:50.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 90998 31.39 MB/sec execute 37 sec latency 93.671 ms 2024-09-21T01:44:50.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 20320 6.59 MB/sec execute 39 sec latency 115.162 ms 2024-09-21T01:44:51.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 92122 32.73 MB/sec execute 38 sec latency 182.167 ms 2024-09-21T01:44:51.937 INFO:tasks.workunit.client.1.smithi090.stdout: 10 20500 6.81 MB/sec execute 40 sec latency 134.338 ms 2024-09-21T01:44:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:51 smithi045 ceph-mon[28361]: pgmap v712: 129 pgs: 129 active+clean; 405 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.6 MiB/s wr, 193 op/s 2024-09-21T01:44:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:51 smithi090 ceph-mon[35142]: pgmap v712: 129 pgs: 129 active+clean; 405 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.6 MiB/s wr, 193 op/s 2024-09-21T01:44:52.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 93005 33.06 MB/sec execute 39 sec latency 333.933 ms 2024-09-21T01:44:52.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 20720 7.28 MB/sec execute 41 sec latency 136.754 ms 2024-09-21T01:44:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:44:53.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 93143 32.32 MB/sec execute 40 sec latency 391.029 ms 2024-09-21T01:44:53.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 20925 7.39 MB/sec execute 42 sec latency 173.659 ms 2024-09-21T01:44:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:53 smithi045 ceph-mon[28361]: pgmap v713: 129 pgs: 129 active+clean; 410 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-09-21T01:44:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:53 smithi090 ceph-mon[35142]: pgmap v713: 129 pgs: 129 active+clean; 410 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.2 MiB/s wr, 206 op/s 2024-09-21T01:44:54.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 94542 32.99 MB/sec execute 41 sec latency 105.588 ms 2024-09-21T01:44:54.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21074 7.45 MB/sec execute 43 sec latency 119.623 ms 2024-09-21T01:44:55.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 95959 34.52 MB/sec execute 42 sec latency 102.712 ms 2024-09-21T01:44:55.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:55 smithi045 ceph-mon[28361]: pgmap v714: 129 pgs: 129 active+clean; 411 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.8 MiB/s wr, 171 op/s 2024-09-21T01:44:55.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21112 7.35 MB/sec execute 44 sec latency 87.665 ms 2024-09-21T01:44:56.222 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:55 smithi090 ceph-mon[35142]: pgmap v714: 129 pgs: 129 active+clean; 411 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.8 MiB/s wr, 171 op/s 2024-09-21T01:44:56.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 96583 34.45 MB/sec execute 43 sec latency 175.421 ms 2024-09-21T01:44:56.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21173 7.27 MB/sec execute 45 sec latency 91.108 ms 2024-09-21T01:44:57.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 97449 34.76 MB/sec execute 44 sec latency 84.220 ms 2024-09-21T01:44:57.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21298 7.43 MB/sec execute 46 sec latency 119.357 ms 2024-09-21T01:44:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:57 smithi045 ceph-mon[28361]: pgmap v715: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-09-21T01:44:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:57 smithi090 ceph-mon[35142]: pgmap v715: 129 pgs: 129 active+clean; 425 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.8 MiB/s wr, 170 op/s 2024-09-21T01:44:58.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 98134 34.34 MB/sec execute 45 sec latency 408.863 ms 2024-09-21T01:44:58.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21430 7.57 MB/sec execute 47 sec latency 241.489 ms 2024-09-21T01:44:59.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 98419 34.10 MB/sec execute 46 sec latency 496.930 ms 2024-09-21T01:44:59.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21553 7.52 MB/sec execute 48 sec latency 171.944 ms 2024-09-21T01:45:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:44:59 smithi045 ceph-mon[28361]: pgmap v716: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.7 MiB/s wr, 132 op/s 2024-09-21T01:45:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:44:59 smithi090 ceph-mon[35142]: pgmap v716: 129 pgs: 129 active+clean; 429 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.7 MiB/s wr, 132 op/s 2024-09-21T01:45:00.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 98776 34.07 MB/sec execute 47 sec latency 399.108 ms 2024-09-21T01:45:00.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21636 7.46 MB/sec execute 49 sec latency 212.530 ms 2024-09-21T01:45:01.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 99032 33.63 MB/sec execute 48 sec latency 447.995 ms 2024-09-21T01:45:01.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21713 7.32 MB/sec execute 50 sec latency 160.318 ms 2024-09-21T01:45:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:01 smithi045 ceph-mon[28361]: pgmap v717: 129 pgs: 129 active+clean; 430 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.2 MiB/s wr, 123 op/s 2024-09-21T01:45:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:01 smithi090 ceph-mon[35142]: pgmap v717: 129 pgs: 129 active+clean; 430 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.2 MiB/s wr, 123 op/s 2024-09-21T01:45:02.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 99128 33.09 MB/sec execute 49 sec latency 497.068 ms 2024-09-21T01:45:02.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21771 7.21 MB/sec execute 51 sec latency 241.440 ms 2024-09-21T01:45:03.697 INFO:tasks.workunit.client.0.smithi045.stdout: 10 99506 32.82 MB/sec execute 50 sec latency 442.564 ms 2024-09-21T01:45:03.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21847 7.08 MB/sec execute 52 sec latency 224.648 ms 2024-09-21T01:45:04.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:03 smithi045 ceph-mon[28361]: pgmap v718: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.4 MiB/s wr, 117 op/s 2024-09-21T01:45:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:03 smithi090 ceph-mon[35142]: pgmap v718: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.4 MiB/s wr, 117 op/s 2024-09-21T01:45:04.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100042 32.76 MB/sec execute 51 sec latency 473.809 ms 2024-09-21T01:45:04.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21910 6.95 MB/sec execute 53 sec latency 250.054 ms 2024-09-21T01:45:05.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100052 32.13 MB/sec execute 52 sec latency 519.703 ms 2024-09-21T01:45:05.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 21985 6.84 MB/sec execute 54 sec latency 190.165 ms 2024-09-21T01:45:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:05 smithi045 ceph-mon[28361]: pgmap v719: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 96 op/s 2024-09-21T01:45:06.223 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:05 smithi090 ceph-mon[35142]: pgmap v719: 129 pgs: 129 active+clean; 435 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 96 op/s 2024-09-21T01:45:06.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100090 31.55 MB/sec execute 53 sec latency 492.095 ms 2024-09-21T01:45:06.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22074 6.76 MB/sec execute 55 sec latency 207.517 ms 2024-09-21T01:45:07.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100101 30.97 MB/sec execute 54 sec latency 408.359 ms 2024-09-21T01:45:07.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22181 6.68 MB/sec execute 56 sec latency 150.513 ms 2024-09-21T01:45:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:07 smithi045 ceph-mon[28361]: pgmap v720: 129 pgs: 129 active+clean; 437 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.2 MiB/s wr, 124 op/s 2024-09-21T01:45:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:07 smithi090 ceph-mon[35142]: pgmap v720: 129 pgs: 129 active+clean; 437 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.2 MiB/s wr, 124 op/s 2024-09-21T01:45:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:08.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100150 30.43 MB/sec execute 55 sec latency 391.017 ms 2024-09-21T01:45:08.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22252 6.61 MB/sec execute 57 sec latency 231.407 ms 2024-09-21T01:45:09.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100167 29.89 MB/sec execute 56 sec latency 475.501 ms 2024-09-21T01:45:09.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22347 6.56 MB/sec execute 58 sec latency 218.899 ms 2024-09-21T01:45:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:09 smithi045 ceph-mon[28361]: pgmap v721: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.9 MiB/s wr, 92 op/s 2024-09-21T01:45:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:09 smithi090 ceph-mon[35142]: pgmap v721: 129 pgs: 129 active+clean; 434 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.9 MiB/s wr, 92 op/s 2024-09-21T01:45:10.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100179 29.37 MB/sec execute 57 sec latency 450.647 ms 2024-09-21T01:45:10.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22447 6.57 MB/sec execute 59 sec latency 186.948 ms 2024-09-21T01:45:11.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100255 28.90 MB/sec execute 58 sec latency 392.270 ms 2024-09-21T01:45:11.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22557 6.57 MB/sec execute 60 sec latency 181.577 ms 2024-09-21T01:45:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:11 smithi045 ceph-mon[28361]: pgmap v722: 129 pgs: 129 active+clean; 428 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.5 MiB/s wr, 101 op/s 2024-09-21T01:45:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:11 smithi090 ceph-mon[35142]: pgmap v722: 129 pgs: 129 active+clean; 428 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.5 MiB/s wr, 101 op/s 2024-09-21T01:45:12.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100338 28.44 MB/sec execute 59 sec latency 358.829 ms 2024-09-21T01:45:12.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22662 6.57 MB/sec execute 61 sec latency 201.435 ms 2024-09-21T01:45:13.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 100841 28.60 MB/sec execute 60 sec latency 467.422 ms 2024-09-21T01:45:13.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22772 6.61 MB/sec execute 62 sec latency 222.054 ms 2024-09-21T01:45:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:13 smithi045 ceph-mon[28361]: pgmap v723: 129 pgs: 129 active+clean; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.5 MiB/s wr, 148 op/s 2024-09-21T01:45:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:13 smithi090 ceph-mon[35142]: pgmap v723: 129 pgs: 129 active+clean; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.5 MiB/s wr, 148 op/s 2024-09-21T01:45:14.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 101299 28.27 MB/sec execute 61 sec latency 487.612 ms 2024-09-21T01:45:14.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 22924 6.66 MB/sec execute 63 sec latency 135.092 ms 2024-09-21T01:45:15.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 102658 29.22 MB/sec execute 62 sec latency 101.699 ms 2024-09-21T01:45:15.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23081 6.57 MB/sec execute 64 sec latency 79.104 ms 2024-09-21T01:45:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:15 smithi045 ceph-mon[28361]: pgmap v724: 129 pgs: 129 active+clean; 416 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-09-21T01:45:16.223 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:15 smithi090 ceph-mon[35142]: pgmap v724: 129 pgs: 129 active+clean; 416 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 1.8 MiB/s wr, 120 op/s 2024-09-21T01:45:16.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 103648 29.58 MB/sec execute 63 sec latency 159.236 ms 2024-09-21T01:45:16.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23260 6.50 MB/sec execute 65 sec latency 110.818 ms 2024-09-21T01:45:17.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 103837 29.19 MB/sec execute 64 sec latency 174.123 ms 2024-09-21T01:45:17.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23411 6.46 MB/sec execute 66 sec latency 93.226 ms 2024-09-21T01:45:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:17 smithi045 ceph-mon[28361]: pgmap v725: 129 pgs: 129 active+clean; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-09-21T01:45:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:17 smithi090 ceph-mon[35142]: pgmap v725: 129 pgs: 129 active+clean; 418 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-09-21T01:45:18.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 105251 29.67 MB/sec execute 65 sec latency 172.858 ms 2024-09-21T01:45:18.938 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23560 6.42 MB/sec execute 67 sec latency 118.072 ms 2024-09-21T01:45:19.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 105958 30.12 MB/sec execute 66 sec latency 390.189 ms 2024-09-21T01:45:19.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23726 6.46 MB/sec execute 68 sec latency 81.547 ms 2024-09-21T01:45:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:19 smithi045 ceph-mon[28361]: pgmap v726: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-09-21T01:45:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:19 smithi090 ceph-mon[35142]: pgmap v726: 129 pgs: 129 active+clean; 417 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 1.7 MiB/s wr, 138 op/s 2024-09-21T01:45:20.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 106181 29.88 MB/sec execute 67 sec latency 482.463 ms 2024-09-21T01:45:20.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23835 6.51 MB/sec execute 69 sec latency 140.205 ms 2024-09-21T01:45:21.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 106506 29.69 MB/sec execute 68 sec latency 425.428 ms 2024-09-21T01:45:21.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 23944 6.53 MB/sec execute 70 sec latency 163.927 ms 2024-09-21T01:45:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:21 smithi090 ceph-mon[35142]: pgmap v727: 129 pgs: 129 active+clean; 409 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-09-21T01:45:22.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:21 smithi045 ceph-mon[28361]: pgmap v727: 129 pgs: 129 active+clean; 409 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-09-21T01:45:22.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 107133 29.76 MB/sec execute 69 sec latency 476.229 ms 2024-09-21T01:45:22.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24143 6.67 MB/sec execute 71 sec latency 155.987 ms 2024-09-21T01:45:23.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:23.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:23.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 107174 29.35 MB/sec execute 70 sec latency 450.168 ms 2024-09-21T01:45:23.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24292 6.82 MB/sec execute 72 sec latency 304.031 ms 2024-09-21T01:45:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:23 smithi090 ceph-mon[35142]: pgmap v728: 129 pgs: 129 active+clean; 404 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.6 MiB/s wr, 173 op/s 2024-09-21T01:45:24.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:23 smithi045 ceph-mon[28361]: pgmap v728: 129 pgs: 129 active+clean; 404 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.6 MiB/s wr, 173 op/s 2024-09-21T01:45:24.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 107189 28.94 MB/sec execute 71 sec latency 443.582 ms 2024-09-21T01:45:24.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24455 6.87 MB/sec execute 73 sec latency 236.922 ms 2024-09-21T01:45:25.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 107211 28.54 MB/sec execute 72 sec latency 417.119 ms 2024-09-21T01:45:25.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24581 6.85 MB/sec execute 74 sec latency 209.174 ms 2024-09-21T01:45:26.224 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:25 smithi090 ceph-mon[35142]: pgmap v729: 129 pgs: 129 active+clean; 401 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-09-21T01:45:26.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:25 smithi045 ceph-mon[28361]: pgmap v729: 129 pgs: 129 active+clean; 401 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.2 MiB/s wr, 124 op/s 2024-09-21T01:45:26.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 107255 28.17 MB/sec execute 73 sec latency 400.581 ms 2024-09-21T01:45:26.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24638 6.82 MB/sec execute 75 sec latency 167.828 ms 2024-09-21T01:45:27.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 107838 28.29 MB/sec execute 74 sec latency 397.725 ms 2024-09-21T01:45:27.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24707 6.82 MB/sec execute 76 sec latency 89.190 ms 2024-09-21T01:45:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:27 smithi090 ceph-mon[35142]: pgmap v730: 129 pgs: 129 active+clean; 416 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-09-21T01:45:28.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:27 smithi045 ceph-mon[28361]: pgmap v730: 129 pgs: 129 active+clean; 416 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.6 MiB/s wr, 157 op/s 2024-09-21T01:45:28.698 INFO:tasks.workunit.client.0.smithi045.stdout: 10 109404 29.00 MB/sec execute 75 sec latency 109.780 ms 2024-09-21T01:45:28.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24809 6.84 MB/sec execute 77 sec latency 118.313 ms 2024-09-21T01:45:29.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 110688 29.53 MB/sec execute 76 sec latency 105.798 ms 2024-09-21T01:45:29.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 24954 6.92 MB/sec execute 78 sec latency 122.083 ms 2024-09-21T01:45:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:29 smithi090 ceph-mon[35142]: pgmap v731: 129 pgs: 129 active+clean; 419 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.3 MiB/s wr, 121 op/s 2024-09-21T01:45:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:45:30.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:29 smithi045 ceph-mon[28361]: pgmap v731: 129 pgs: 129 active+clean; 419 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.3 MiB/s wr, 121 op/s 2024-09-21T01:45:30.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:29 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:45:30.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 110810 29.19 MB/sec execute 77 sec latency 221.143 ms 2024-09-21T01:45:30.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25097 6.98 MB/sec execute 79 sec latency 147.389 ms 2024-09-21T01:45:31.224 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:45:31.224 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:45:31.224 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:30 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:45:31.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:45:31.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:45:31.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:30 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:45:31.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 112029 29.50 MB/sec execute 78 sec latency 93.241 ms 2024-09-21T01:45:31.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25208 6.96 MB/sec execute 80 sec latency 120.387 ms 2024-09-21T01:45:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:31 smithi045 ceph-mon[28361]: pgmap v732: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.0 MiB/s wr, 135 op/s 2024-09-21T01:45:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:31 smithi090 ceph-mon[35142]: pgmap v732: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.0 MiB/s wr, 135 op/s 2024-09-21T01:45:32.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 112459 29.35 MB/sec execute 79 sec latency 417.395 ms 2024-09-21T01:45:32.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25276 6.88 MB/sec execute 81 sec latency 191.529 ms 2024-09-21T01:45:33.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:32 smithi090 ceph-mon[35142]: pgmap v733: 129 pgs: 129 active+clean; 424 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.1 MiB/s wr, 109 op/s 2024-09-21T01:45:33.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:32 smithi045 ceph-mon[28361]: pgmap v733: 129 pgs: 129 active+clean; 424 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.1 MiB/s wr, 109 op/s 2024-09-21T01:45:33.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 112876 29.52 MB/sec execute 80 sec latency 505.166 ms 2024-09-21T01:45:33.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25362 6.82 MB/sec execute 82 sec latency 164.127 ms 2024-09-21T01:45:34.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 113035 29.24 MB/sec execute 81 sec latency 534.761 ms 2024-09-21T01:45:34.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25432 6.74 MB/sec execute 83 sec latency 238.800 ms 2024-09-21T01:45:35.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:35 smithi045 ceph-mon[28361]: pgmap v734: 129 pgs: 129 active+clean; 424 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.6 MiB/s wr, 92 op/s 2024-09-21T01:45:35.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 113276 29.08 MB/sec execute 82 sec latency 483.919 ms 2024-09-21T01:45:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:35 smithi090 ceph-mon[35142]: pgmap v734: 129 pgs: 129 active+clean; 424 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.6 MiB/s wr, 92 op/s 2024-09-21T01:45:35.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25507 6.67 MB/sec execute 84 sec latency 219.160 ms 2024-09-21T01:45:36.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 114038 29.22 MB/sec execute 83 sec latency 538.680 ms 2024-09-21T01:45:36.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25592 6.62 MB/sec execute 85 sec latency 238.565 ms 2024-09-21T01:45:37.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:37 smithi045 ceph-mon[28361]: pgmap v735: 129 pgs: 129 active+clean; 425 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.2 MiB/s wr, 119 op/s 2024-09-21T01:45:37.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 114271 29.01 MB/sec execute 84 sec latency 562.318 ms 2024-09-21T01:45:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:37 smithi090 ceph-mon[35142]: pgmap v735: 129 pgs: 129 active+clean; 425 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.2 MiB/s wr, 119 op/s 2024-09-21T01:45:37.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25709 6.57 MB/sec execute 86 sec latency 209.445 ms 2024-09-21T01:45:38.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:38 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:38.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 114451 28.72 MB/sec execute 85 sec latency 217.319 ms 2024-09-21T01:45:38.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:38 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:38.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25855 6.58 MB/sec execute 87 sec latency 117.572 ms 2024-09-21T01:45:39.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:39 smithi045 ceph-mon[28361]: pgmap v736: 129 pgs: 129 active+clean; 425 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 109 op/s 2024-09-21T01:45:39.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 115411 28.92 MB/sec execute 86 sec latency 330.234 ms 2024-09-21T01:45:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:39 smithi090 ceph-mon[35142]: pgmap v736: 129 pgs: 129 active+clean; 425 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.1 MiB/s wr, 109 op/s 2024-09-21T01:45:39.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 25985 6.56 MB/sec execute 88 sec latency 102.173 ms 2024-09-21T01:45:40.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 116837 29.61 MB/sec execute 87 sec latency 100.312 ms 2024-09-21T01:45:40.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26144 6.61 MB/sec execute 89 sec latency 107.066 ms 2024-09-21T01:45:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:41 smithi090 ceph-mon[35142]: pgmap v737: 129 pgs: 129 active+clean; 420 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.1 MiB/s wr, 142 op/s 2024-09-21T01:45:41.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:41 smithi045 ceph-mon[28361]: pgmap v737: 129 pgs: 129 active+clean; 420 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.1 MiB/s wr, 142 op/s 2024-09-21T01:45:41.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 117884 29.88 MB/sec execute 88 sec latency 95.184 ms 2024-09-21T01:45:41.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26314 6.65 MB/sec execute 90 sec latency 94.067 ms 2024-09-21T01:45:42.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 118557 30.02 MB/sec execute 89 sec latency 179.442 ms 2024-09-21T01:45:42.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26456 6.71 MB/sec execute 91 sec latency 169.415 ms 2024-09-21T01:45:43.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:43 smithi045 ceph-mon[28361]: pgmap v738: 129 pgs: 129 active+clean; 408 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.9 MiB/s wr, 147 op/s 2024-09-21T01:45:43.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 118890 29.71 MB/sec execute 90 sec latency 552.776 ms 2024-09-21T01:45:43.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:43 smithi090 ceph-mon[35142]: pgmap v738: 129 pgs: 129 active+clean; 408 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.9 MiB/s wr, 147 op/s 2024-09-21T01:45:43.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26566 6.65 MB/sec execute 92 sec latency 173.822 ms 2024-09-21T01:45:44.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 119049 29.47 MB/sec execute 91 sec latency 618.153 ms 2024-09-21T01:45:44.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26699 6.59 MB/sec execute 93 sec latency 208.236 ms 2024-09-21T01:45:45.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:45 smithi045 ceph-mon[28361]: pgmap v739: 129 pgs: 129 active+clean; 407 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.4 MiB/s wr, 145 op/s 2024-09-21T01:45:45.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 119432 29.28 MB/sec execute 92 sec latency 535.666 ms 2024-09-21T01:45:45.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:45 smithi090 ceph-mon[35142]: pgmap v739: 129 pgs: 129 active+clean; 407 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.4 MiB/s wr, 145 op/s 2024-09-21T01:45:45.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26845 6.54 MB/sec execute 94 sec latency 178.441 ms 2024-09-21T01:45:46.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 119849 29.40 MB/sec execute 93 sec latency 453.174 ms 2024-09-21T01:45:46.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 26958 6.50 MB/sec execute 95 sec latency 164.009 ms 2024-09-21T01:45:47.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:47 smithi045 ceph-mon[28361]: pgmap v740: 129 pgs: 129 active+clean; 409 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 2.8 MiB/s wr, 190 op/s 2024-09-21T01:45:47.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 120356 29.43 MB/sec execute 94 sec latency 464.509 ms 2024-09-21T01:45:47.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:47 smithi090 ceph-mon[35142]: pgmap v740: 129 pgs: 129 active+clean; 409 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 2.8 MiB/s wr, 190 op/s 2024-09-21T01:45:47.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 27109 6.48 MB/sec execute 96 sec latency 158.867 ms 2024-09-21T01:45:48.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 121374 29.68 MB/sec execute 95 sec latency 88.904 ms 2024-09-21T01:45:48.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 27285 6.51 MB/sec execute 97 sec latency 72.713 ms 2024-09-21T01:45:49.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:49 smithi045 ceph-mon[28361]: pgmap v741: 129 pgs: 129 active+clean; 410 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-09-21T01:45:49.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 121716 29.51 MB/sec execute 96 sec latency 156.465 ms 2024-09-21T01:45:49.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:49 smithi090 ceph-mon[35142]: pgmap v741: 129 pgs: 129 active+clean; 410 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 173 op/s 2024-09-21T01:45:49.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 27425 6.55 MB/sec execute 98 sec latency 92.855 ms 2024-09-21T01:45:50.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 123217 29.95 MB/sec execute 97 sec latency 110.374 ms 2024-09-21T01:45:50.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 27583 6.62 MB/sec execute 99 sec latency 126.192 ms 2024-09-21T01:45:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:51 smithi090 ceph-mon[35142]: pgmap v742: 129 pgs: 129 active+clean; 413 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-09-21T01:45:51.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:51 smithi045 ceph-mon[28361]: pgmap v742: 129 pgs: 129 active+clean; 413 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 2.4 MiB/s wr, 175 op/s 2024-09-21T01:45:51.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 124773 30.65 MB/sec execute 98 sec latency 124.216 ms 2024-09-21T01:45:51.939 INFO:tasks.workunit.client.1.smithi090.stdout: 10 27775 6.70 MB/sec execute 100 sec latency 113.867 ms 2024-09-21T01:45:52.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 124980 30.42 MB/sec execute 99 sec latency 95.477 ms 2024-09-21T01:45:52.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28008 6.86 MB/sec execute 101 sec latency 109.082 ms 2024-09-21T01:45:53.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:53 smithi045 ceph-mon[28361]: pgmap v743: 129 pgs: 129 active+clean; 421 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-09-21T01:45:53.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:53 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:53.699 INFO:tasks.workunit.client.0.smithi045.stdout: 10 125809 30.55 MB/sec execute 100 sec latency 109.005 ms 2024-09-21T01:45:53.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:53 smithi090 ceph-mon[35142]: pgmap v743: 129 pgs: 129 active+clean; 421 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.9 MiB/s wr, 158 op/s 2024-09-21T01:45:53.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:53 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:45:53.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28158 6.87 MB/sec execute 102 sec latency 151.863 ms 2024-09-21T01:45:54.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 127082 30.97 MB/sec execute 101 sec latency 130.619 ms 2024-09-21T01:45:54.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28220 6.86 MB/sec execute 103 sec latency 122.013 ms 2024-09-21T01:45:55.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:55 smithi045 ceph-mon[28361]: pgmap v744: 129 pgs: 129 active+clean; 422 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.6 MiB/s wr, 129 op/s 2024-09-21T01:45:55.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 128410 31.38 MB/sec execute 102 sec latency 124.193 ms 2024-09-21T01:45:55.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:55 smithi090 ceph-mon[35142]: pgmap v744: 129 pgs: 129 active+clean; 422 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.6 MiB/s wr, 129 op/s 2024-09-21T01:45:55.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28299 6.86 MB/sec execute 104 sec latency 108.036 ms 2024-09-21T01:45:56.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 128551 31.10 MB/sec execute 103 sec latency 92.576 ms 2024-09-21T01:45:56.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28410 6.90 MB/sec execute 105 sec latency 125.618 ms 2024-09-21T01:45:57.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:57 smithi045 ceph-mon[28361]: pgmap v745: 129 pgs: 129 active+clean; 435 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 161 op/s 2024-09-21T01:45:57.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 129638 31.29 MB/sec execute 104 sec latency 135.390 ms 2024-09-21T01:45:57.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:57 smithi090 ceph-mon[35142]: pgmap v745: 129 pgs: 129 active+clean; 435 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.8 MiB/s wr, 161 op/s 2024-09-21T01:45:57.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28561 6.98 MB/sec execute 106 sec latency 110.906 ms 2024-09-21T01:45:58.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 130933 31.75 MB/sec execute 105 sec latency 117.402 ms 2024-09-21T01:45:58.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28677 6.95 MB/sec execute 107 sec latency 123.785 ms 2024-09-21T01:45:59.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:45:59 smithi045 ceph-mon[28361]: pgmap v746: 129 pgs: 129 active+clean; 436 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-09-21T01:45:59.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 131978 32.00 MB/sec execute 106 sec latency 225.964 ms 2024-09-21T01:45:59.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:45:59 smithi090 ceph-mon[35142]: pgmap v746: 129 pgs: 129 active+clean; 436 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.7 MiB/s wr, 116 op/s 2024-09-21T01:45:59.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28788 6.93 MB/sec execute 108 sec latency 132.526 ms 2024-09-21T01:46:00.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 132167 31.74 MB/sec execute 107 sec latency 84.547 ms 2024-09-21T01:46:00.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28882 6.88 MB/sec execute 109 sec latency 132.132 ms 2024-09-21T01:46:01.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:01 smithi090 ceph-mon[35142]: pgmap v747: 129 pgs: 129 active+clean; 440 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.6 MiB/s wr, 136 op/s 2024-09-21T01:46:01.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:01 smithi045 ceph-mon[28361]: pgmap v747: 129 pgs: 129 active+clean; 440 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.6 MiB/s wr, 136 op/s 2024-09-21T01:46:01.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 133763 32.12 MB/sec execute 108 sec latency 111.279 ms 2024-09-21T01:46:01.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 28987 6.83 MB/sec execute 110 sec latency 122.690 ms 2024-09-21T01:46:02.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 135408 32.83 MB/sec execute 109 sec latency 129.857 ms 2024-09-21T01:46:02.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29102 6.79 MB/sec execute 111 sec latency 155.455 ms 2024-09-21T01:46:03.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:03 smithi045 ceph-mon[28361]: pgmap v748: 129 pgs: 129 active+clean; 442 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.5 MiB/s wr, 116 op/s 2024-09-21T01:46:03.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 135558 32.58 MB/sec execute 110 sec latency 303.065 ms 2024-09-21T01:46:03.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:03 smithi090 ceph-mon[35142]: pgmap v748: 129 pgs: 129 active+clean; 442 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.5 MiB/s wr, 116 op/s 2024-09-21T01:46:03.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29207 6.74 MB/sec execute 112 sec latency 166.674 ms 2024-09-21T01:46:04.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 135952 32.46 MB/sec execute 111 sec latency 133.983 ms 2024-09-21T01:46:04.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29355 6.75 MB/sec execute 113 sec latency 141.157 ms 2024-09-21T01:46:05.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:05 smithi045 ceph-mon[28361]: pgmap v749: 129 pgs: 129 active+clean; 442 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.6 MiB/s wr, 97 op/s 2024-09-21T01:46:05.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 137179 32.60 MB/sec execute 112 sec latency 253.998 ms 2024-09-21T01:46:05.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:05 smithi090 ceph-mon[35142]: pgmap v749: 129 pgs: 129 active+clean; 442 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.6 MiB/s wr, 97 op/s 2024-09-21T01:46:05.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29498 6.74 MB/sec execute 114 sec latency 114.738 ms 2024-09-21T01:46:06.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 138955 33.36 MB/sec execute 113 sec latency 132.761 ms 2024-09-21T01:46:06.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29649 6.78 MB/sec execute 115 sec latency 96.030 ms 2024-09-21T01:46:07.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:07 smithi045 ceph-mon[28361]: pgmap v750: 129 pgs: 129 active+clean; 433 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.1 MiB/s wr, 160 op/s 2024-09-21T01:46:07.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 139173 33.13 MB/sec execute 114 sec latency 92.911 ms 2024-09-21T01:46:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:07 smithi090 ceph-mon[35142]: pgmap v750: 129 pgs: 129 active+clean; 433 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.1 MiB/s wr, 160 op/s 2024-09-21T01:46:07.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29812 6.80 MB/sec execute 116 sec latency 100.335 ms 2024-09-21T01:46:08.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:08 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:08.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 140084 33.26 MB/sec execute 115 sec latency 92.879 ms 2024-09-21T01:46:08.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:08 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:08.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 29948 6.80 MB/sec execute 117 sec latency 217.048 ms 2024-09-21T01:46:09.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:09 smithi045 ceph-mon[28361]: pgmap v751: 129 pgs: 129 active+clean; 418 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.3 MiB/s wr, 143 op/s 2024-09-21T01:46:09.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 140399 33.02 MB/sec execute 116 sec latency 476.977 ms 2024-09-21T01:46:09.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:09 smithi090 ceph-mon[35142]: pgmap v751: 129 pgs: 129 active+clean; 418 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.3 MiB/s wr, 143 op/s 2024-09-21T01:46:09.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30061 6.80 MB/sec execute 118 sec latency 188.292 ms 2024-09-21T01:46:10.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 140772 32.86 MB/sec execute 117 sec latency 566.510 ms 2024-09-21T01:46:10.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30207 6.75 MB/sec execute 119 sec latency 167.714 ms 2024-09-21T01:46:11.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:11 smithi090 ceph-mon[35142]: pgmap v752: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.5 MiB/s wr, 181 op/s 2024-09-21T01:46:11.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:11 smithi045 ceph-mon[28361]: pgmap v752: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.5 MiB/s wr, 181 op/s 2024-09-21T01:46:11.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 141001 32.77 MB/sec execute 118 sec latency 539.409 ms 2024-09-21T01:46:11.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30337 6.71 MB/sec execute 120 sec latency 152.756 ms 2024-09-21T01:46:12.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 141379 32.76 MB/sec execute 119 sec latency 475.662 ms 2024-09-21T01:46:12.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30470 6.68 MB/sec execute 121 sec latency 163.486 ms 2024-09-21T01:46:13.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:13 smithi045 ceph-mon[28361]: pgmap v753: 129 pgs: 129 active+clean; 400 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-09-21T01:46:13.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 141537 32.55 MB/sec execute 120 sec latency 507.411 ms 2024-09-21T01:46:13.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:13 smithi090 ceph-mon[35142]: pgmap v753: 129 pgs: 129 active+clean; 400 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.7 MiB/s wr, 168 op/s 2024-09-21T01:46:13.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30563 6.65 MB/sec execute 122 sec latency 194.458 ms 2024-09-21T01:46:14.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 141681 32.36 MB/sec execute 121 sec latency 459.323 ms 2024-09-21T01:46:14.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30669 6.61 MB/sec execute 123 sec latency 204.246 ms 2024-09-21T01:46:15.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:15 smithi045 ceph-mon[28361]: pgmap v754: 129 pgs: 129 active+clean; 400 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-09-21T01:46:15.700 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142322 32.38 MB/sec execute 122 sec latency 559.926 ms 2024-09-21T01:46:15.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:15 smithi090 ceph-mon[35142]: pgmap v754: 129 pgs: 129 active+clean; 400 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.2 MiB/s wr, 162 op/s 2024-09-21T01:46:15.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30780 6.62 MB/sec execute 124 sec latency 251.955 ms 2024-09-21T01:46:16.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142594 32.24 MB/sec execute 123 sec latency 447.440 ms 2024-09-21T01:46:16.940 INFO:tasks.workunit.client.1.smithi090.stdout: 10 30920 6.65 MB/sec execute 125 sec latency 168.611 ms 2024-09-21T01:46:17.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:17 smithi045 ceph-mon[28361]: pgmap v755: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.6 MiB/s wr, 206 op/s 2024-09-21T01:46:17.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142631 31.99 MB/sec execute 124 sec latency 448.336 ms 2024-09-21T01:46:17.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:17 smithi090 ceph-mon[35142]: pgmap v755: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.6 MiB/s wr, 206 op/s 2024-09-21T01:46:17.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31011 6.65 MB/sec execute 126 sec latency 285.899 ms 2024-09-21T01:46:18.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142648 31.73 MB/sec execute 125 sec latency 433.885 ms 2024-09-21T01:46:18.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31095 6.64 MB/sec execute 127 sec latency 181.019 ms 2024-09-21T01:46:19.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:19 smithi045 ceph-mon[28361]: pgmap v756: 129 pgs: 129 active+clean; 406 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.3 MiB/s wr, 152 op/s 2024-09-21T01:46:19.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142670 31.49 MB/sec execute 126 sec latency 381.718 ms 2024-09-21T01:46:19.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:19 smithi090 ceph-mon[35142]: pgmap v756: 129 pgs: 129 active+clean; 406 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.3 MiB/s wr, 152 op/s 2024-09-21T01:46:19.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31215 6.69 MB/sec execute 128 sec latency 198.595 ms 2024-09-21T01:46:20.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142710 31.25 MB/sec execute 127 sec latency 384.006 ms 2024-09-21T01:46:20.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31325 6.68 MB/sec execute 129 sec latency 195.342 ms 2024-09-21T01:46:21.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:21 smithi090 ceph-mon[35142]: pgmap v757: 129 pgs: 129 active+clean; 408 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-09-21T01:46:21.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:21 smithi045 ceph-mon[28361]: pgmap v757: 129 pgs: 129 active+clean; 408 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.3 MiB/s wr, 137 op/s 2024-09-21T01:46:21.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142720 31.00 MB/sec execute 128 sec latency 475.536 ms 2024-09-21T01:46:21.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31436 6.74 MB/sec execute 130 sec latency 222.829 ms 2024-09-21T01:46:22.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142801 30.78 MB/sec execute 129 sec latency 425.837 ms 2024-09-21T01:46:22.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:22.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:22.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31575 6.77 MB/sec execute 131 sec latency 167.684 ms 2024-09-21T01:46:23.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:23 smithi045 ceph-mon[28361]: pgmap v758: 129 pgs: 129 active+clean; 415 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-09-21T01:46:23.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 142852 30.55 MB/sec execute 130 sec latency 375.154 ms 2024-09-21T01:46:23.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:23 smithi090 ceph-mon[35142]: pgmap v758: 129 pgs: 129 active+clean; 415 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.5 MiB/s wr, 119 op/s 2024-09-21T01:46:23.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31689 6.76 MB/sec execute 132 sec latency 186.517 ms 2024-09-21T01:46:24.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 143295 30.55 MB/sec execute 131 sec latency 442.333 ms 2024-09-21T01:46:24.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31762 6.75 MB/sec execute 133 sec latency 162.983 ms 2024-09-21T01:46:25.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 144245 30.54 MB/sec execute 132 sec latency 446.698 ms 2024-09-21T01:46:25.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:25 smithi090 ceph-mon[35142]: pgmap v759: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.4 MiB/s wr, 99 op/s 2024-09-21T01:46:25.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:25 smithi045 ceph-mon[28361]: pgmap v759: 129 pgs: 129 active+clean; 417 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.4 MiB/s wr, 99 op/s 2024-09-21T01:46:25.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31835 6.75 MB/sec execute 134 sec latency 151.023 ms 2024-09-21T01:46:26.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 145507 30.99 MB/sec execute 133 sec latency 209.493 ms 2024-09-21T01:46:26.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 31923 6.74 MB/sec execute 135 sec latency 112.294 ms 2024-09-21T01:46:27.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 146256 31.04 MB/sec execute 134 sec latency 103.008 ms 2024-09-21T01:46:27.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:27 smithi090 ceph-mon[35142]: pgmap v760: 129 pgs: 129 active+clean; 427 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.3 MiB/s wr, 120 op/s 2024-09-21T01:46:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:27 smithi045 ceph-mon[28361]: pgmap v760: 129 pgs: 129 active+clean; 427 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.3 MiB/s wr, 120 op/s 2024-09-21T01:46:27.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32041 6.76 MB/sec execute 136 sec latency 125.115 ms 2024-09-21T01:46:28.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 147275 31.17 MB/sec execute 135 sec latency 92.046 ms 2024-09-21T01:46:28.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32197 6.82 MB/sec execute 137 sec latency 107.936 ms 2024-09-21T01:46:29.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 148622 31.53 MB/sec execute 136 sec latency 116.809 ms 2024-09-21T01:46:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:29 smithi090 ceph-mon[35142]: pgmap v761: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.1 MiB/s wr, 80 op/s 2024-09-21T01:46:29.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:29 smithi045 ceph-mon[28361]: pgmap v761: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.1 MiB/s wr, 80 op/s 2024-09-21T01:46:29.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32315 6.81 MB/sec execute 138 sec latency 118.550 ms 2024-09-21T01:46:30.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149681 31.73 MB/sec execute 137 sec latency 433.697 ms 2024-09-21T01:46:30.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32390 6.77 MB/sec execute 139 sec latency 236.099 ms 2024-09-21T01:46:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:31 smithi090 ceph-mon[35142]: pgmap v762: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.5 MiB/s wr, 75 op/s 2024-09-21T01:46:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:31 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:46:31.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149686 31.51 MB/sec execute 138 sec latency 598.503 ms 2024-09-21T01:46:31.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:31 smithi045 ceph-mon[28361]: pgmap v762: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.5 MiB/s wr, 75 op/s 2024-09-21T01:46:31.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:31 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:46:31.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32457 6.73 MB/sec execute 140 sec latency 257.225 ms 2024-09-21T01:46:32.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149721 31.29 MB/sec execute 139 sec latency 467.210 ms 2024-09-21T01:46:32.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32514 6.68 MB/sec execute 141 sec latency 212.389 ms 2024-09-21T01:46:33.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149739 31.06 MB/sec execute 140 sec latency 458.960 ms 2024-09-21T01:46:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:33 smithi090 ceph-mon[35142]: pgmap v763: 129 pgs: 129 active+clean; 431 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.9 MiB/s wr, 107 op/s 2024-09-21T01:46:33.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:33 smithi045 ceph-mon[28361]: pgmap v763: 129 pgs: 129 active+clean; 431 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.9 MiB/s wr, 107 op/s 2024-09-21T01:46:33.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32595 6.65 MB/sec execute 142 sec latency 191.395 ms 2024-09-21T01:46:34.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149784 30.85 MB/sec execute 141 sec latency 390.681 ms 2024-09-21T01:46:34.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32674 6.61 MB/sec execute 143 sec latency 210.889 ms 2024-09-21T01:46:35.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149806 30.64 MB/sec execute 142 sec latency 437.411 ms 2024-09-21T01:46:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:35 smithi090 ceph-mon[35142]: pgmap v764: 129 pgs: 129 active+clean; 432 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 MiB/s wr, 81 op/s 2024-09-21T01:46:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:46:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:46:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:46:35.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:35 smithi045 ceph-mon[28361]: pgmap v764: 129 pgs: 129 active+clean; 432 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.2 MiB/s wr, 81 op/s 2024-09-21T01:46:35.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:46:35.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:46:35.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:46:35.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32760 6.58 MB/sec execute 144 sec latency 186.333 ms 2024-09-21T01:46:36.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 149868 30.43 MB/sec execute 143 sec latency 419.537 ms 2024-09-21T01:46:36.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 32874 6.57 MB/sec execute 145 sec latency 179.748 ms 2024-09-21T01:46:37.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 151029 30.59 MB/sec execute 144 sec latency 280.135 ms 2024-09-21T01:46:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:37 smithi090 ceph-mon[35142]: pgmap v765: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.3 MiB/s wr, 97 op/s 2024-09-21T01:46:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:37.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:37 smithi045 ceph-mon[28361]: pgmap v765: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.3 MiB/s wr, 97 op/s 2024-09-21T01:46:37.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:37.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33017 6.55 MB/sec execute 146 sec latency 116.255 ms 2024-09-21T01:46:38.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 152299 30.96 MB/sec execute 145 sec latency 124.423 ms 2024-09-21T01:46:38.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33163 6.57 MB/sec execute 147 sec latency 107.278 ms 2024-09-21T01:46:39.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 153297 31.10 MB/sec execute 146 sec latency 119.047 ms 2024-09-21T01:46:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:39 smithi090 ceph-mon[35142]: pgmap v766: 129 pgs: 129 active+clean; 426 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-09-21T01:46:39.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:39 smithi045 ceph-mon[28361]: pgmap v766: 129 pgs: 129 active+clean; 426 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.8 MiB/s wr, 111 op/s 2024-09-21T01:46:39.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33310 6.58 MB/sec execute 148 sec latency 115.043 ms 2024-09-21T01:46:40.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 153934 31.13 MB/sec execute 147 sec latency 92.429 ms 2024-09-21T01:46:40.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33466 6.59 MB/sec execute 149 sec latency 107.579 ms 2024-09-21T01:46:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:41 smithi090 ceph-mon[35142]: pgmap v767: 129 pgs: 129 active+clean; 423 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.9 MiB/s wr, 124 op/s 2024-09-21T01:46:41.701 INFO:tasks.workunit.client.0.smithi045.stdout: 10 154970 31.15 MB/sec execute 148 sec latency 278.095 ms 2024-09-21T01:46:41.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:41 smithi045 ceph-mon[28361]: pgmap v767: 129 pgs: 129 active+clean; 423 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.9 MiB/s wr, 124 op/s 2024-09-21T01:46:41.941 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33629 6.62 MB/sec execute 150 sec latency 126.828 ms 2024-09-21T01:46:42.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 155519 31.29 MB/sec execute 149 sec latency 432.594 ms 2024-09-21T01:46:42.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33743 6.59 MB/sec execute 151 sec latency 176.578 ms 2024-09-21T01:46:43.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 155784 31.17 MB/sec execute 150 sec latency 435.646 ms 2024-09-21T01:46:43.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:43 smithi090 ceph-mon[35142]: pgmap v768: 129 pgs: 129 active+clean; 415 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-09-21T01:46:43.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:43 smithi045 ceph-mon[28361]: pgmap v768: 129 pgs: 129 active+clean; 415 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-09-21T01:46:44.222 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33858 6.56 MB/sec execute 152 sec latency 216.906 ms 2024-09-21T01:46:44.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 155858 30.99 MB/sec execute 151 sec latency 506.109 ms 2024-09-21T01:46:44.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 33990 6.53 MB/sec execute 153 sec latency 197.566 ms 2024-09-21T01:46:45.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 156503 31.02 MB/sec execute 152 sec latency 412.776 ms 2024-09-21T01:46:45.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:45 smithi090 ceph-mon[35142]: pgmap v769: 129 pgs: 129 active+clean; 416 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.1 MiB/s wr, 146 op/s 2024-09-21T01:46:45.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:45 smithi045 ceph-mon[28361]: pgmap v769: 129 pgs: 129 active+clean; 416 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.1 MiB/s wr, 146 op/s 2024-09-21T01:46:45.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 34100 6.51 MB/sec execute 154 sec latency 172.408 ms 2024-09-21T01:46:46.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 156783 30.92 MB/sec execute 153 sec latency 442.150 ms 2024-09-21T01:46:46.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 34219 6.49 MB/sec execute 155 sec latency 277.376 ms 2024-09-21T01:46:47.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 156954 30.74 MB/sec execute 154 sec latency 152.455 ms 2024-09-21T01:46:47.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:47 smithi090 ceph-mon[35142]: pgmap v770: 129 pgs: 129 active+clean; 416 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.1 MiB/s wr, 169 op/s 2024-09-21T01:46:47.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:47 smithi045 ceph-mon[28361]: pgmap v770: 129 pgs: 129 active+clean; 416 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.1 MiB/s wr, 169 op/s 2024-09-21T01:46:47.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 34398 6.51 MB/sec execute 156 sec latency 99.968 ms 2024-09-21T01:46:48.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 157959 30.85 MB/sec execute 155 sec latency 284.832 ms 2024-09-21T01:46:48.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 34548 6.55 MB/sec execute 157 sec latency 156.518 ms 2024-09-21T01:46:49.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 159365 31.21 MB/sec execute 156 sec latency 105.134 ms 2024-09-21T01:46:49.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:49 smithi090 ceph-mon[35142]: pgmap v771: 129 pgs: 129 active+clean; 418 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-09-21T01:46:49.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:49 smithi045 ceph-mon[28361]: pgmap v771: 129 pgs: 129 active+clean; 418 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 182 op/s 2024-09-21T01:46:49.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 34706 6.58 MB/sec execute 158 sec latency 113.125 ms 2024-09-21T01:46:50.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 160377 31.35 MB/sec execute 157 sec latency 99.914 ms 2024-09-21T01:46:50.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 34886 6.62 MB/sec execute 159 sec latency 116.720 ms 2024-09-21T01:46:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:51 smithi090 ceph-mon[35142]: pgmap v772: 129 pgs: 129 active+clean; 423 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.6 MiB/s wr, 162 op/s 2024-09-21T01:46:51.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 160609 31.19 MB/sec execute 158 sec latency 87.225 ms 2024-09-21T01:46:51.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:51 smithi045 ceph-mon[28361]: pgmap v772: 129 pgs: 129 active+clean; 423 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.6 MiB/s wr, 162 op/s 2024-09-21T01:46:51.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35091 6.71 MB/sec execute 160 sec latency 99.861 ms 2024-09-21T01:46:52.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 162117 31.43 MB/sec execute 159 sec latency 128.351 ms 2024-09-21T01:46:52.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:52.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:46:52.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35254 6.74 MB/sec execute 161 sec latency 113.059 ms 2024-09-21T01:46:53.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 163863 31.95 MB/sec execute 160 sec latency 117.006 ms 2024-09-21T01:46:53.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:53 smithi090 ceph-mon[35142]: pgmap v773: 129 pgs: 129 active+clean; 429 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-09-21T01:46:53.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:53 smithi045 ceph-mon[28361]: pgmap v773: 129 pgs: 129 active+clean; 429 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.9 MiB/s wr, 163 op/s 2024-09-21T01:46:53.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35337 6.74 MB/sec execute 162 sec latency 97.943 ms 2024-09-21T01:46:54.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 163987 31.77 MB/sec execute 161 sec latency 129.975 ms 2024-09-21T01:46:54.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35419 6.74 MB/sec execute 163 sec latency 91.561 ms 2024-09-21T01:46:55.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 164802 31.87 MB/sec execute 162 sec latency 117.941 ms 2024-09-21T01:46:55.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:55 smithi090 ceph-mon[35142]: pgmap v774: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-09-21T01:46:55.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:55 smithi045 ceph-mon[28361]: pgmap v774: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.7 MiB/s wr, 124 op/s 2024-09-21T01:46:55.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35501 6.72 MB/sec execute 164 sec latency 103.545 ms 2024-09-21T01:46:56.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 166098 32.09 MB/sec execute 163 sec latency 221.377 ms 2024-09-21T01:46:56.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35641 6.77 MB/sec execute 165 sec latency 123.951 ms 2024-09-21T01:46:57.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 167451 32.34 MB/sec execute 164 sec latency 94.856 ms 2024-09-21T01:46:57.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:57 smithi090 ceph-mon[35142]: pgmap v775: 129 pgs: 129 active+clean; 441 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 135 op/s 2024-09-21T01:46:57.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:57 smithi045 ceph-mon[28361]: pgmap v775: 129 pgs: 129 active+clean; 441 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 135 op/s 2024-09-21T01:46:57.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35783 6.78 MB/sec execute 166 sec latency 107.778 ms 2024-09-21T01:46:58.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 167652 32.18 MB/sec execute 165 sec latency 106.755 ms 2024-09-21T01:46:58.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 35900 6.77 MB/sec execute 167 sec latency 123.920 ms 2024-09-21T01:46:59.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 168942 32.33 MB/sec execute 166 sec latency 372.148 ms 2024-09-21T01:46:59.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:46:59 smithi090 ceph-mon[35142]: pgmap v776: 129 pgs: 129 active+clean; 444 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.2 MiB/s wr, 116 op/s 2024-09-21T01:46:59.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:46:59 smithi045 ceph-mon[28361]: pgmap v776: 129 pgs: 129 active+clean; 444 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.2 MiB/s wr, 116 op/s 2024-09-21T01:46:59.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36000 6.74 MB/sec execute 168 sec latency 141.486 ms 2024-09-21T01:47:00.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 170048 32.61 MB/sec execute 167 sec latency 107.300 ms 2024-09-21T01:47:00.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36103 6.72 MB/sec execute 169 sec latency 144.276 ms 2024-09-21T01:47:01.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:01 smithi090 ceph-mon[35142]: pgmap v777: 129 pgs: 129 active+clean; 445 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 102 op/s 2024-09-21T01:47:01.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 171021 32.72 MB/sec execute 168 sec latency 119.372 ms 2024-09-21T01:47:01.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:01 smithi045 ceph-mon[28361]: pgmap v777: 129 pgs: 129 active+clean; 445 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 102 op/s 2024-09-21T01:47:01.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36207 6.68 MB/sec execute 170 sec latency 150.648 ms 2024-09-21T01:47:02.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 171090 32.53 MB/sec execute 169 sec latency 358.858 ms 2024-09-21T01:47:02.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36320 6.66 MB/sec execute 171 sec latency 175.906 ms 2024-09-21T01:47:03.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 172115 32.63 MB/sec execute 170 sec latency 123.911 ms 2024-09-21T01:47:03.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:03 smithi090 ceph-mon[35142]: pgmap v778: 129 pgs: 129 active+clean; 448 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.4 MiB/s wr, 126 op/s 2024-09-21T01:47:03.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:03 smithi045 ceph-mon[28361]: pgmap v778: 129 pgs: 129 active+clean; 448 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.4 MiB/s wr, 126 op/s 2024-09-21T01:47:03.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36460 6.66 MB/sec execute 172 sec latency 121.641 ms 2024-09-21T01:47:04.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 173480 32.93 MB/sec execute 171 sec latency 117.945 ms 2024-09-21T01:47:04.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36601 6.66 MB/sec execute 173 sec latency 115.102 ms 2024-09-21T01:47:05.702 INFO:tasks.workunit.client.0.smithi045.stdout: 10 174560 33.07 MB/sec execute 172 sec latency 92.124 ms 2024-09-21T01:47:05.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:05 smithi090 ceph-mon[35142]: pgmap v779: 129 pgs: 129 active+clean; 447 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.7 MiB/s wr, 109 op/s 2024-09-21T01:47:05.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:05 smithi045 ceph-mon[28361]: pgmap v779: 129 pgs: 129 active+clean; 447 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.7 MiB/s wr, 109 op/s 2024-09-21T01:47:05.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36746 6.68 MB/sec execute 174 sec latency 97.020 ms 2024-09-21T01:47:06.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 174792 32.91 MB/sec execute 173 sec latency 84.941 ms 2024-09-21T01:47:06.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 36892 6.66 MB/sec execute 175 sec latency 106.225 ms 2024-09-21T01:47:07.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 176054 33.04 MB/sec execute 174 sec latency 196.661 ms 2024-09-21T01:47:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:07 smithi090 ceph-mon[35142]: pgmap v780: 129 pgs: 129 active+clean; 420 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.9 MiB/s wr, 146 op/s 2024-09-21T01:47:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:07.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:07 smithi045 ceph-mon[28361]: pgmap v780: 129 pgs: 129 active+clean; 420 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.9 MiB/s wr, 146 op/s 2024-09-21T01:47:07.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:07.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37028 6.66 MB/sec execute 176 sec latency 271.581 ms 2024-09-21T01:47:08.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 176293 32.93 MB/sec execute 175 sec latency 484.071 ms 2024-09-21T01:47:08.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37154 6.69 MB/sec execute 177 sec latency 179.196 ms 2024-09-21T01:47:09.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 176633 32.93 MB/sec execute 176 sec latency 526.234 ms 2024-09-21T01:47:09.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:09 smithi090 ceph-mon[35142]: pgmap v781: 129 pgs: 129 active+clean; 406 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.1 MiB/s wr, 165 op/s 2024-09-21T01:47:09.860 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:09 smithi045 ceph-mon[28361]: pgmap v781: 129 pgs: 129 active+clean; 406 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.1 MiB/s wr, 165 op/s 2024-09-21T01:47:09.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37283 6.67 MB/sec execute 178 sec latency 152.792 ms 2024-09-21T01:47:10.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 176913 32.85 MB/sec execute 177 sec latency 572.134 ms 2024-09-21T01:47:10.942 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37417 6.64 MB/sec execute 179 sec latency 185.994 ms 2024-09-21T01:47:11.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:11 smithi090 ceph-mon[35142]: pgmap v782: 129 pgs: 129 active+clean; 406 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-09-21T01:47:11.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 177099 32.72 MB/sec execute 178 sec latency 549.862 ms 2024-09-21T01:47:11.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:11 smithi045 ceph-mon[28361]: pgmap v782: 129 pgs: 129 active+clean; 406 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 2.9 MiB/s wr, 171 op/s 2024-09-21T01:47:11.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37544 6.61 MB/sec execute 180 sec latency 189.268 ms 2024-09-21T01:47:12.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 177677 32.72 MB/sec execute 179 sec latency 425.746 ms 2024-09-21T01:47:12.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37648 6.60 MB/sec execute 181 sec latency 261.698 ms 2024-09-21T01:47:13.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 177782 32.57 MB/sec execute 180 sec latency 487.712 ms 2024-09-21T01:47:13.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:13 smithi090 ceph-mon[35142]: pgmap v783: 129 pgs: 129 active+clean; 407 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 191 op/s 2024-09-21T01:47:13.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:13 smithi045 ceph-mon[28361]: pgmap v783: 129 pgs: 129 active+clean; 407 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 191 op/s 2024-09-21T01:47:13.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37751 6.57 MB/sec execute 182 sec latency 244.839 ms 2024-09-21T01:47:14.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178051 32.47 MB/sec execute 181 sec latency 394.083 ms 2024-09-21T01:47:14.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 37908 6.58 MB/sec execute 183 sec latency 122.158 ms 2024-09-21T01:47:15.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178087 32.30 MB/sec execute 182 sec latency 384.048 ms 2024-09-21T01:47:15.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:15 smithi090 ceph-mon[35142]: pgmap v784: 129 pgs: 129 active+clean; 407 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 1.9 MiB/s wr, 154 op/s 2024-09-21T01:47:15.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:15 smithi045 ceph-mon[28361]: pgmap v784: 129 pgs: 129 active+clean; 407 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 1.9 MiB/s wr, 154 op/s 2024-09-21T01:47:15.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38033 6.60 MB/sec execute 184 sec latency 171.901 ms 2024-09-21T01:47:16.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178104 32.12 MB/sec execute 183 sec latency 508.488 ms 2024-09-21T01:47:16.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38116 6.60 MB/sec execute 185 sec latency 217.794 ms 2024-09-21T01:47:17.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178143 31.95 MB/sec execute 184 sec latency 362.984 ms 2024-09-21T01:47:17.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:17 smithi090 ceph-mon[35142]: pgmap v785: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 2.4 MiB/s wr, 191 op/s 2024-09-21T01:47:17.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:17 smithi045 ceph-mon[28361]: pgmap v785: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 2.4 MiB/s wr, 191 op/s 2024-09-21T01:47:17.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38237 6.61 MB/sec execute 186 sec latency 224.765 ms 2024-09-21T01:47:18.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178170 31.78 MB/sec execute 185 sec latency 408.878 ms 2024-09-21T01:47:18.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38352 6.64 MB/sec execute 187 sec latency 208.222 ms 2024-09-21T01:47:19.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178204 31.62 MB/sec execute 186 sec latency 417.187 ms 2024-09-21T01:47:19.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:19 smithi090 ceph-mon[35142]: pgmap v786: 129 pgs: 129 active+clean; 414 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-09-21T01:47:19.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:19 smithi045 ceph-mon[28361]: pgmap v786: 129 pgs: 129 active+clean; 414 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.4 MiB/s wr, 160 op/s 2024-09-21T01:47:19.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38458 6.64 MB/sec execute 188 sec latency 168.953 ms 2024-09-21T01:47:20.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178251 31.46 MB/sec execute 187 sec latency 417.337 ms 2024-09-21T01:47:20.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38587 6.67 MB/sec execute 189 sec latency 264.722 ms 2024-09-21T01:47:21.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178323 31.30 MB/sec execute 188 sec latency 375.260 ms 2024-09-21T01:47:21.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:21 smithi090 ceph-mon[35142]: pgmap v787: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-21T01:47:21.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:21 smithi045 ceph-mon[28361]: pgmap v787: 129 pgs: 129 active+clean; 418 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-21T01:47:21.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38692 6.68 MB/sec execute 190 sec latency 201.437 ms 2024-09-21T01:47:22.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 178715 31.27 MB/sec execute 189 sec latency 417.388 ms 2024-09-21T01:47:22.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:22.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:22.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38829 6.70 MB/sec execute 191 sec latency 170.481 ms 2024-09-21T01:47:23.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 179110 31.20 MB/sec execute 190 sec latency 455.661 ms 2024-09-21T01:47:23.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:23 smithi090 ceph-mon[35142]: pgmap v788: 129 pgs: 129 active+clean; 428 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.1 MiB/s wr, 132 op/s 2024-09-21T01:47:23.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:23 smithi045 ceph-mon[28361]: pgmap v788: 129 pgs: 129 active+clean; 428 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.1 MiB/s wr, 132 op/s 2024-09-21T01:47:23.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38880 6.69 MB/sec execute 192 sec latency 212.971 ms 2024-09-21T01:47:24.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 179439 31.08 MB/sec execute 191 sec latency 497.421 ms 2024-09-21T01:47:24.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 38955 6.69 MB/sec execute 193 sec latency 150.048 ms 2024-09-21T01:47:25.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 180643 31.33 MB/sec execute 192 sec latency 258.496 ms 2024-09-21T01:47:25.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:25 smithi045 ceph-mon[28361]: pgmap v789: 129 pgs: 129 active+clean; 429 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.8 MiB/s wr, 97 op/s 2024-09-21T01:47:25.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39031 6.67 MB/sec execute 194 sec latency 107.972 ms 2024-09-21T01:47:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:25 smithi090 ceph-mon[35142]: pgmap v789: 129 pgs: 129 active+clean; 429 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.8 MiB/s wr, 97 op/s 2024-09-21T01:47:26.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 181652 31.45 MB/sec execute 193 sec latency 125.208 ms 2024-09-21T01:47:26.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39132 6.67 MB/sec execute 195 sec latency 130.418 ms 2024-09-21T01:47:27.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 181983 31.36 MB/sec execute 194 sec latency 93.399 ms 2024-09-21T01:47:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:27 smithi045 ceph-mon[28361]: pgmap v790: 129 pgs: 129 active+clean; 434 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.4 MiB/s wr, 102 op/s 2024-09-21T01:47:27.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39274 6.72 MB/sec execute 196 sec latency 161.041 ms 2024-09-21T01:47:27.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:27 smithi090 ceph-mon[35142]: pgmap v790: 129 pgs: 129 active+clean; 434 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.4 MiB/s wr, 102 op/s 2024-09-21T01:47:28.703 INFO:tasks.workunit.client.0.smithi045.stdout: 10 183403 31.52 MB/sec execute 195 sec latency 173.152 ms 2024-09-21T01:47:28.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39395 6.70 MB/sec execute 197 sec latency 104.052 ms 2024-09-21T01:47:29.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 184870 31.86 MB/sec execute 196 sec latency 99.931 ms 2024-09-21T01:47:29.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:29 smithi045 ceph-mon[28361]: pgmap v791: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.4 MiB/s wr, 87 op/s 2024-09-21T01:47:29.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39498 6.68 MB/sec execute 198 sec latency 161.578 ms 2024-09-21T01:47:29.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:29 smithi090 ceph-mon[35142]: pgmap v791: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.4 MiB/s wr, 87 op/s 2024-09-21T01:47:30.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185183 31.78 MB/sec execute 197 sec latency 357.382 ms 2024-09-21T01:47:30.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39593 6.66 MB/sec execute 199 sec latency 128.519 ms 2024-09-21T01:47:31.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185196 31.62 MB/sec execute 198 sec latency 409.028 ms 2024-09-21T01:47:31.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:31 smithi045 ceph-mon[28361]: pgmap v792: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.5 MiB/s wr, 83 op/s 2024-09-21T01:47:31.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39670 6.63 MB/sec execute 200 sec latency 186.034 ms 2024-09-21T01:47:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:31 smithi090 ceph-mon[35142]: pgmap v792: 129 pgs: 129 active+clean; 437 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.5 MiB/s wr, 83 op/s 2024-09-21T01:47:32.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185233 31.47 MB/sec execute 199 sec latency 495.677 ms 2024-09-21T01:47:32.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39748 6.60 MB/sec execute 201 sec latency 199.025 ms 2024-09-21T01:47:33.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185261 31.31 MB/sec execute 200 sec latency 450.923 ms 2024-09-21T01:47:33.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:33 smithi045 ceph-mon[28361]: pgmap v793: 129 pgs: 129 active+clean; 440 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.8 MiB/s wr, 116 op/s 2024-09-21T01:47:33.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39821 6.58 MB/sec execute 202 sec latency 196.607 ms 2024-09-21T01:47:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:33 smithi090 ceph-mon[35142]: pgmap v793: 129 pgs: 129 active+clean; 440 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.8 MiB/s wr, 116 op/s 2024-09-21T01:47:34.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185281 31.16 MB/sec execute 201 sec latency 467.998 ms 2024-09-21T01:47:34.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39914 6.56 MB/sec execute 203 sec latency 205.472 ms 2024-09-21T01:47:35.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185334 31.01 MB/sec execute 202 sec latency 450.929 ms 2024-09-21T01:47:35.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 39999 6.55 MB/sec execute 204 sec latency 195.360 ms 2024-09-21T01:47:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:35 smithi045 ceph-mon[28361]: pgmap v794: 129 pgs: 129 active+clean; 439 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.8 MiB/s wr, 85 op/s 2024-09-21T01:47:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:47:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:47:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:47:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:35 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:47:36.228 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:35 smithi090 ceph-mon[35142]: pgmap v794: 129 pgs: 129 active+clean; 439 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.8 MiB/s wr, 85 op/s 2024-09-21T01:47:36.228 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:47:36.228 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:47:36.229 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:47:36.229 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:35 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:47:36.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185408 30.87 MB/sec execute 203 sec latency 390.987 ms 2024-09-21T01:47:36.943 INFO:tasks.workunit.client.1.smithi090.stdout: 10 40091 6.53 MB/sec execute 205 sec latency 181.877 ms 2024-09-21T01:47:37.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 185804 30.84 MB/sec execute 204 sec latency 524.465 ms 2024-09-21T01:47:37.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 40210 6.54 MB/sec execute 206 sec latency 174.262 ms 2024-09-21T01:47:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:37 smithi045 ceph-mon[28361]: pgmap v795: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-09-21T01:47:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:37 smithi090 ceph-mon[35142]: pgmap v795: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.1 MiB/s wr, 111 op/s 2024-09-21T01:47:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:38.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 186701 30.85 MB/sec execute 205 sec latency 501.869 ms 2024-09-21T01:47:38.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 40357 6.55 MB/sec execute 207 sec latency 203.792 ms 2024-09-21T01:47:39.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 187778 31.08 MB/sec execute 206 sec latency 175.975 ms 2024-09-21T01:47:39.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 40501 6.55 MB/sec execute 208 sec latency 113.118 ms 2024-09-21T01:47:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:39 smithi045 ceph-mon[28361]: pgmap v796: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-09-21T01:47:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:39 smithi090 ceph-mon[35142]: pgmap v796: 129 pgs: 129 active+clean; 427 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.7 MiB/s wr, 121 op/s 2024-09-21T01:47:40.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 188744 31.17 MB/sec execute 207 sec latency 200.756 ms 2024-09-21T01:47:40.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 40682 6.58 MB/sec execute 209 sec latency 91.794 ms 2024-09-21T01:47:41.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 188977 31.05 MB/sec execute 208 sec latency 92.191 ms 2024-09-21T01:47:41.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 40849 6.56 MB/sec execute 210 sec latency 97.867 ms 2024-09-21T01:47:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:41 smithi045 ceph-mon[28361]: pgmap v797: 129 pgs: 129 active+clean; 422 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.6 MiB/s wr, 114 op/s 2024-09-21T01:47:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:41 smithi090 ceph-mon[35142]: pgmap v797: 129 pgs: 129 active+clean; 422 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.6 MiB/s wr, 114 op/s 2024-09-21T01:47:42.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 190251 31.17 MB/sec execute 209 sec latency 316.140 ms 2024-09-21T01:47:42.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41009 6.54 MB/sec execute 211 sec latency 95.805 ms 2024-09-21T01:47:43.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 191215 31.37 MB/sec execute 210 sec latency 333.000 ms 2024-09-21T01:47:43.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41138 6.53 MB/sec execute 212 sec latency 176.337 ms 2024-09-21T01:47:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:43 smithi045 ceph-mon[28361]: pgmap v798: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-09-21T01:47:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:43 smithi090 ceph-mon[35142]: pgmap v798: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.5 MiB/s wr, 160 op/s 2024-09-21T01:47:44.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 191682 31.34 MB/sec execute 211 sec latency 458.825 ms 2024-09-21T01:47:44.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41260 6.51 MB/sec execute 213 sec latency 154.138 ms 2024-09-21T01:47:45.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 191961 31.26 MB/sec execute 212 sec latency 484.751 ms 2024-09-21T01:47:45.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41398 6.52 MB/sec execute 214 sec latency 164.234 ms 2024-09-21T01:47:46.076 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:45 smithi045 ceph-mon[28361]: pgmap v799: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:47:46.228 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:45 smithi090 ceph-mon[35142]: pgmap v799: 129 pgs: 129 active+clean; 421 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:47:46.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 192260 31.19 MB/sec execute 213 sec latency 391.865 ms 2024-09-21T01:47:46.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41515 6.52 MB/sec execute 215 sec latency 171.053 ms 2024-09-21T01:47:47.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 192269 31.04 MB/sec execute 214 sec latency 433.941 ms 2024-09-21T01:47:47.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41639 6.54 MB/sec execute 216 sec latency 175.442 ms 2024-09-21T01:47:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:47 smithi045 ceph-mon[28361]: pgmap v800: 129 pgs: 129 active+clean; 424 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.2 MiB/s wr, 170 op/s 2024-09-21T01:47:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:47 smithi090 ceph-mon[35142]: pgmap v800: 129 pgs: 129 active+clean; 424 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.2 MiB/s wr, 170 op/s 2024-09-21T01:47:48.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 192334 30.90 MB/sec execute 215 sec latency 433.690 ms 2024-09-21T01:47:48.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41777 6.54 MB/sec execute 217 sec latency 184.752 ms 2024-09-21T01:47:49.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 193362 30.99 MB/sec execute 216 sec latency 107.971 ms 2024-09-21T01:47:49.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 41948 6.58 MB/sec execute 218 sec latency 96.062 ms 2024-09-21T01:47:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:49 smithi045 ceph-mon[28361]: pgmap v801: 129 pgs: 129 active+clean; 425 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-09-21T01:47:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:49 smithi090 ceph-mon[35142]: pgmap v801: 129 pgs: 129 active+clean; 425 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.2 MiB/s wr, 157 op/s 2024-09-21T01:47:50.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 194680 31.22 MB/sec execute 217 sec latency 144.361 ms 2024-09-21T01:47:50.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42109 6.63 MB/sec execute 219 sec latency 88.039 ms 2024-09-21T01:47:51.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 195816 31.35 MB/sec execute 218 sec latency 120.231 ms 2024-09-21T01:47:51.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42276 6.65 MB/sec execute 220 sec latency 101.690 ms 2024-09-21T01:47:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:51 smithi045 ceph-mon[28361]: pgmap v802: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.7 MiB/s wr, 163 op/s 2024-09-21T01:47:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:51 smithi090 ceph-mon[35142]: pgmap v802: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.7 MiB/s wr, 163 op/s 2024-09-21T01:47:52.704 INFO:tasks.workunit.client.0.smithi045.stdout: 10 195959 31.22 MB/sec execute 219 sec latency 107.460 ms 2024-09-21T01:47:52.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42421 6.67 MB/sec execute 221 sec latency 116.389 ms 2024-09-21T01:47:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:47:53.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 196913 31.29 MB/sec execute 220 sec latency 201.513 ms 2024-09-21T01:47:53.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42481 6.66 MB/sec execute 222 sec latency 213.888 ms 2024-09-21T01:47:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:53 smithi045 ceph-mon[28361]: pgmap v803: 129 pgs: 129 active+clean; 441 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-09-21T01:47:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:53 smithi090 ceph-mon[35142]: pgmap v803: 129 pgs: 129 active+clean; 441 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.3 MiB/s wr, 162 op/s 2024-09-21T01:47:54.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 197319 31.20 MB/sec execute 221 sec latency 539.728 ms 2024-09-21T01:47:54.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42546 6.65 MB/sec execute 223 sec latency 159.113 ms 2024-09-21T01:47:55.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 197614 31.15 MB/sec execute 222 sec latency 556.436 ms 2024-09-21T01:47:55.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42615 6.64 MB/sec execute 224 sec latency 223.826 ms 2024-09-21T01:47:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:55 smithi045 ceph-mon[28361]: pgmap v804: 129 pgs: 129 active+clean; 443 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.0 MiB/s wr, 111 op/s 2024-09-21T01:47:56.229 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:55 smithi090 ceph-mon[35142]: pgmap v804: 129 pgs: 129 active+clean; 443 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.0 MiB/s wr, 111 op/s 2024-09-21T01:47:56.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 197872 31.12 MB/sec execute 223 sec latency 623.305 ms 2024-09-21T01:47:56.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42687 6.63 MB/sec execute 225 sec latency 230.148 ms 2024-09-21T01:47:57.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 198142 31.07 MB/sec execute 224 sec latency 498.852 ms 2024-09-21T01:47:57.944 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42825 6.67 MB/sec execute 226 sec latency 151.535 ms 2024-09-21T01:47:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:57 smithi045 ceph-mon[28361]: pgmap v805: 129 pgs: 129 active+clean; 450 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 140 op/s 2024-09-21T01:47:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:57 smithi090 ceph-mon[35142]: pgmap v805: 129 pgs: 129 active+clean; 450 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 140 op/s 2024-09-21T01:47:58.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 198309 30.98 MB/sec execute 225 sec latency 535.522 ms 2024-09-21T01:47:58.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 42932 6.65 MB/sec execute 227 sec latency 193.619 ms 2024-09-21T01:47:59.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 198685 30.93 MB/sec execute 226 sec latency 408.949 ms 2024-09-21T01:47:59.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43013 6.64 MB/sec execute 228 sec latency 217.199 ms 2024-09-21T01:48:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:47:59 smithi045 ceph-mon[28361]: pgmap v806: 129 pgs: 129 active+clean; 450 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.7 MiB/s wr, 98 op/s 2024-09-21T01:48:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:47:59 smithi090 ceph-mon[35142]: pgmap v806: 129 pgs: 129 active+clean; 450 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.7 MiB/s wr, 98 op/s 2024-09-21T01:48:00.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 199360 30.95 MB/sec execute 227 sec latency 355.000 ms 2024-09-21T01:48:00.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43113 6.62 MB/sec execute 229 sec latency 276.263 ms 2024-09-21T01:48:01.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 199562 30.83 MB/sec execute 228 sec latency 100.497 ms 2024-09-21T01:48:01.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43219 6.60 MB/sec execute 230 sec latency 153.413 ms 2024-09-21T01:48:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:01 smithi045 ceph-mon[28361]: pgmap v807: 129 pgs: 129 active+clean; 453 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.1 MiB/s wr, 113 op/s 2024-09-21T01:48:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:01 smithi090 ceph-mon[35142]: pgmap v807: 129 pgs: 129 active+clean; 453 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.1 MiB/s wr, 113 op/s 2024-09-21T01:48:02.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 200902 30.96 MB/sec execute 229 sec latency 107.558 ms 2024-09-21T01:48:02.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43332 6.58 MB/sec execute 231 sec latency 104.660 ms 2024-09-21T01:48:03.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 202219 31.22 MB/sec execute 230 sec latency 110.862 ms 2024-09-21T01:48:03.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43469 6.58 MB/sec execute 232 sec latency 141.424 ms 2024-09-21T01:48:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:03 smithi045 ceph-mon[28361]: pgmap v808: 129 pgs: 129 active+clean; 453 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.9 MiB/s wr, 98 op/s 2024-09-21T01:48:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:03 smithi090 ceph-mon[35142]: pgmap v808: 129 pgs: 129 active+clean; 453 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.9 MiB/s wr, 98 op/s 2024-09-21T01:48:04.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 202943 31.24 MB/sec execute 231 sec latency 114.395 ms 2024-09-21T01:48:04.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43607 6.57 MB/sec execute 233 sec latency 144.597 ms 2024-09-21T01:48:05.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 203301 31.18 MB/sec execute 232 sec latency 100.688 ms 2024-09-21T01:48:05.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43751 6.58 MB/sec execute 234 sec latency 128.949 ms 2024-09-21T01:48:06.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:05 smithi045 ceph-mon[28361]: pgmap v809: 129 pgs: 129 active+clean; 453 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.9 MiB/s wr, 85 op/s 2024-09-21T01:48:06.229 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:05 smithi090 ceph-mon[35142]: pgmap v809: 129 pgs: 129 active+clean; 453 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.9 MiB/s wr, 85 op/s 2024-09-21T01:48:06.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 204642 31.31 MB/sec execute 233 sec latency 137.394 ms 2024-09-21T01:48:06.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 43903 6.58 MB/sec execute 235 sec latency 131.909 ms 2024-09-21T01:48:07.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 205777 31.51 MB/sec execute 234 sec latency 131.147 ms 2024-09-21T01:48:07.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 44061 6.59 MB/sec execute 236 sec latency 93.076 ms 2024-09-21T01:48:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:07 smithi045 ceph-mon[28361]: pgmap v810: 129 pgs: 129 active+clean; 438 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 144 op/s 2024-09-21T01:48:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:07 smithi090 ceph-mon[35142]: pgmap v810: 129 pgs: 129 active+clean; 438 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 144 op/s 2024-09-21T01:48:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:08.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 206488 31.53 MB/sec execute 235 sec latency 110.619 ms 2024-09-21T01:48:08.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 44249 6.61 MB/sec execute 237 sec latency 91.532 ms 2024-09-21T01:48:09.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 206643 31.41 MB/sec execute 236 sec latency 292.206 ms 2024-09-21T01:48:09.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 44406 6.59 MB/sec execute 238 sec latency 83.551 ms 2024-09-21T01:48:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:09 smithi045 ceph-mon[28361]: pgmap v811: 129 pgs: 129 active+clean; 420 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.0 MiB/s wr, 145 op/s 2024-09-21T01:48:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:09 smithi090 ceph-mon[35142]: pgmap v811: 129 pgs: 129 active+clean; 420 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.0 MiB/s wr, 145 op/s 2024-09-21T01:48:10.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 207833 31.50 MB/sec execute 237 sec latency 210.829 ms 2024-09-21T01:48:10.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 44554 6.58 MB/sec execute 239 sec latency 112.314 ms 2024-09-21T01:48:11.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 208988 31.70 MB/sec execute 238 sec latency 170.464 ms 2024-09-21T01:48:11.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 44723 6.57 MB/sec execute 240 sec latency 121.838 ms 2024-09-21T01:48:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:11 smithi045 ceph-mon[28361]: pgmap v812: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.0 MiB/s wr, 159 op/s 2024-09-21T01:48:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:11 smithi090 ceph-mon[35142]: pgmap v812: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.0 MiB/s wr, 159 op/s 2024-09-21T01:48:12.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 210000 31.79 MB/sec execute 239 sec latency 186.176 ms 2024-09-21T01:48:12.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 44891 6.57 MB/sec execute 241 sec latency 130.598 ms 2024-09-21T01:48:13.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 210154 31.68 MB/sec execute 240 sec latency 106.188 ms 2024-09-21T01:48:13.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45062 6.58 MB/sec execute 242 sec latency 84.797 ms 2024-09-21T01:48:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:13 smithi045 ceph-mon[28361]: pgmap v813: 129 pgs: 129 active+clean; 414 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.8 MiB/s wr, 183 op/s 2024-09-21T01:48:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:13 smithi090 ceph-mon[35142]: pgmap v813: 129 pgs: 129 active+clean; 414 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.8 MiB/s wr, 183 op/s 2024-09-21T01:48:14.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 211291 31.76 MB/sec execute 241 sec latency 140.192 ms 2024-09-21T01:48:14.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45236 6.61 MB/sec execute 243 sec latency 93.617 ms 2024-09-21T01:48:15.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 212433 31.94 MB/sec execute 242 sec latency 242.472 ms 2024-09-21T01:48:15.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45364 6.63 MB/sec execute 244 sec latency 157.429 ms 2024-09-21T01:48:16.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:15 smithi045 ceph-mon[28361]: pgmap v814: 129 pgs: 129 active+clean; 414 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.3 MiB/s wr, 177 op/s 2024-09-21T01:48:16.229 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:15 smithi090 ceph-mon[35142]: pgmap v814: 129 pgs: 129 active+clean; 414 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.3 MiB/s wr, 177 op/s 2024-09-21T01:48:16.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 213501 32.05 MB/sec execute 243 sec latency 426.045 ms 2024-09-21T01:48:16.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45524 6.64 MB/sec execute 245 sec latency 209.004 ms 2024-09-21T01:48:17.705 INFO:tasks.workunit.client.0.smithi045.stdout: 10 213617 31.93 MB/sec execute 244 sec latency 244.481 ms 2024-09-21T01:48:17.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45689 6.69 MB/sec execute 246 sec latency 113.806 ms 2024-09-21T01:48:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:17 smithi045 ceph-mon[28361]: pgmap v815: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 211 op/s 2024-09-21T01:48:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:17 smithi090 ceph-mon[35142]: pgmap v815: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 211 op/s 2024-09-21T01:48:18.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 214293 31.94 MB/sec execute 245 sec latency 105.007 ms 2024-09-21T01:48:18.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45870 6.71 MB/sec execute 247 sec latency 90.941 ms 2024-09-21T01:48:19.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 215657 32.10 MB/sec execute 246 sec latency 129.580 ms 2024-09-21T01:48:19.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 45999 6.72 MB/sec execute 248 sec latency 130.086 ms 2024-09-21T01:48:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:19 smithi045 ceph-mon[28361]: pgmap v816: 129 pgs: 129 active+clean; 428 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.1 MiB/s wr, 163 op/s 2024-09-21T01:48:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:19 smithi090 ceph-mon[35142]: pgmap v816: 129 pgs: 129 active+clean; 428 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.1 MiB/s wr, 163 op/s 2024-09-21T01:48:20.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217013 32.29 MB/sec execute 247 sec latency 201.997 ms 2024-09-21T01:48:20.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46095 6.73 MB/sec execute 249 sec latency 120.058 ms 2024-09-21T01:48:21.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217156 32.19 MB/sec execute 248 sec latency 364.557 ms 2024-09-21T01:48:21.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46168 6.72 MB/sec execute 250 sec latency 185.512 ms 2024-09-21T01:48:22.108 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:21 smithi045 ceph-mon[28361]: pgmap v817: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.7 MiB/s wr, 160 op/s 2024-09-21T01:48:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:21 smithi090 ceph-mon[35142]: pgmap v817: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.7 MiB/s wr, 160 op/s 2024-09-21T01:48:22.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217182 32.06 MB/sec execute 249 sec latency 450.534 ms 2024-09-21T01:48:22.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46225 6.71 MB/sec execute 251 sec latency 190.464 ms 2024-09-21T01:48:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:23.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217212 31.93 MB/sec execute 250 sec latency 402.388 ms 2024-09-21T01:48:23.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46310 6.70 MB/sec execute 252 sec latency 194.209 ms 2024-09-21T01:48:24.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:23 smithi045 ceph-mon[28361]: pgmap v818: 129 pgs: 129 active+clean; 440 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.1 MiB/s wr, 153 op/s 2024-09-21T01:48:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:23 smithi090 ceph-mon[35142]: pgmap v818: 129 pgs: 129 active+clean; 440 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.1 MiB/s wr, 153 op/s 2024-09-21T01:48:24.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217246 31.81 MB/sec execute 251 sec latency 431.200 ms 2024-09-21T01:48:24.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46407 6.72 MB/sec execute 253 sec latency 214.050 ms 2024-09-21T01:48:25.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217295 31.69 MB/sec execute 252 sec latency 417.381 ms 2024-09-21T01:48:25.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46495 6.70 MB/sec execute 254 sec latency 223.503 ms 2024-09-21T01:48:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:25 smithi045 ceph-mon[28361]: pgmap v819: 129 pgs: 129 active+clean; 440 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.6 MiB/s wr, 101 op/s 2024-09-21T01:48:26.230 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:25 smithi090 ceph-mon[35142]: pgmap v819: 129 pgs: 129 active+clean; 440 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.6 MiB/s wr, 101 op/s 2024-09-21T01:48:26.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217348 31.57 MB/sec execute 253 sec latency 483.890 ms 2024-09-21T01:48:26.945 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46579 6.69 MB/sec execute 255 sec latency 167.744 ms 2024-09-21T01:48:27.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 217749 31.55 MB/sec execute 254 sec latency 336.594 ms 2024-09-21T01:48:27.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46655 6.67 MB/sec execute 256 sec latency 283.919 ms 2024-09-21T01:48:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:27 smithi045 ceph-mon[28361]: pgmap v820: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.6 MiB/s wr, 126 op/s 2024-09-21T01:48:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:27 smithi090 ceph-mon[35142]: pgmap v820: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.6 MiB/s wr, 126 op/s 2024-09-21T01:48:28.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 218042 31.48 MB/sec execute 255 sec latency 425.582 ms 2024-09-21T01:48:28.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46744 6.65 MB/sec execute 257 sec latency 155.923 ms 2024-09-21T01:48:29.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 218376 31.39 MB/sec execute 256 sec latency 544.545 ms 2024-09-21T01:48:29.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46825 6.63 MB/sec execute 258 sec latency 205.603 ms 2024-09-21T01:48:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:29 smithi045 ceph-mon[28361]: pgmap v821: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 92 op/s 2024-09-21T01:48:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:29 smithi090 ceph-mon[35142]: pgmap v821: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 92 op/s 2024-09-21T01:48:30.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 218681 31.31 MB/sec execute 257 sec latency 523.071 ms 2024-09-21T01:48:30.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 46911 6.61 MB/sec execute 259 sec latency 210.413 ms 2024-09-21T01:48:31.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 219107 31.31 MB/sec execute 258 sec latency 492.699 ms 2024-09-21T01:48:31.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47021 6.61 MB/sec execute 260 sec latency 156.684 ms 2024-09-21T01:48:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:31 smithi045 ceph-mon[28361]: pgmap v822: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.4 MiB/s wr, 106 op/s 2024-09-21T01:48:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:31 smithi090 ceph-mon[35142]: pgmap v822: 129 pgs: 129 active+clean; 444 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.4 MiB/s wr, 106 op/s 2024-09-21T01:48:32.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 219367 31.27 MB/sec execute 259 sec latency 503.097 ms 2024-09-21T01:48:32.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47116 6.59 MB/sec execute 261 sec latency 246.541 ms 2024-09-21T01:48:33.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 219552 31.20 MB/sec execute 260 sec latency 605.401 ms 2024-09-21T01:48:33.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47230 6.59 MB/sec execute 262 sec latency 221.627 ms 2024-09-21T01:48:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:33 smithi045 ceph-mon[28361]: pgmap v823: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.0 MiB/s wr, 112 op/s 2024-09-21T01:48:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:33 smithi090 ceph-mon[35142]: pgmap v823: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.0 MiB/s wr, 112 op/s 2024-09-21T01:48:34.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 219974 31.17 MB/sec execute 261 sec latency 473.704 ms 2024-09-21T01:48:34.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47335 6.59 MB/sec execute 263 sec latency 219.487 ms 2024-09-21T01:48:35.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 220281 31.12 MB/sec execute 262 sec latency 601.532 ms 2024-09-21T01:48:35.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47428 6.57 MB/sec execute 264 sec latency 184.012 ms 2024-09-21T01:48:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:35 smithi045 ceph-mon[28361]: pgmap v824: 129 pgs: 129 active+clean; 440 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.3 MiB/s wr, 103 op/s 2024-09-21T01:48:36.230 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:35 smithi090 ceph-mon[35142]: pgmap v824: 129 pgs: 129 active+clean; 440 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.3 MiB/s wr, 103 op/s 2024-09-21T01:48:36.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 220502 31.05 MB/sec execute 263 sec latency 511.843 ms 2024-09-21T01:48:36.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47548 6.57 MB/sec execute 265 sec latency 186.599 ms 2024-09-21T01:48:37.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:48:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:48:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:48:37.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:36 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:48:37.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:48:37.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:48:37.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:48:37.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:36 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:48:37.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 220664 30.96 MB/sec execute 264 sec latency 562.385 ms 2024-09-21T01:48:37.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47679 6.56 MB/sec execute 266 sec latency 187.818 ms 2024-09-21T01:48:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:37 smithi045 ceph-mon[28361]: pgmap v825: 129 pgs: 129 active+clean; 431 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-09-21T01:48:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:37 smithi090 ceph-mon[35142]: pgmap v825: 129 pgs: 129 active+clean; 431 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.7 MiB/s wr, 132 op/s 2024-09-21T01:48:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:38.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 220686 30.84 MB/sec execute 265 sec latency 445.038 ms 2024-09-21T01:48:38.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47840 6.59 MB/sec execute 267 sec latency 196.857 ms 2024-09-21T01:48:39.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 220918 30.75 MB/sec execute 266 sec latency 290.378 ms 2024-09-21T01:48:39.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 47997 6.57 MB/sec execute 268 sec latency 77.988 ms 2024-09-21T01:48:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:39 smithi045 ceph-mon[28361]: pgmap v826: 129 pgs: 129 active+clean; 428 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.1 MiB/s wr, 135 op/s 2024-09-21T01:48:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:39 smithi090 ceph-mon[35142]: pgmap v826: 129 pgs: 129 active+clean; 428 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.1 MiB/s wr, 135 op/s 2024-09-21T01:48:40.706 INFO:tasks.workunit.client.0.smithi045.stdout: 10 222237 30.85 MB/sec execute 267 sec latency 124.997 ms 2024-09-21T01:48:40.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48150 6.56 MB/sec execute 269 sec latency 97.358 ms 2024-09-21T01:48:41.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 223742 31.13 MB/sec execute 268 sec latency 114.638 ms 2024-09-21T01:48:41.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48305 6.55 MB/sec execute 270 sec latency 118.834 ms 2024-09-21T01:48:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:41 smithi045 ceph-mon[28361]: pgmap v827: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-21T01:48:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:41 smithi090 ceph-mon[35142]: pgmap v827: 129 pgs: 129 active+clean; 426 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-21T01:48:42.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 224229 31.09 MB/sec execute 269 sec latency 376.140 ms 2024-09-21T01:48:42.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48476 6.56 MB/sec execute 271 sec latency 93.181 ms 2024-09-21T01:48:43.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 224380 30.99 MB/sec execute 270 sec latency 104.716 ms 2024-09-21T01:48:43.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48654 6.57 MB/sec execute 272 sec latency 98.182 ms 2024-09-21T01:48:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:43 smithi045 ceph-mon[28361]: pgmap v828: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-09-21T01:48:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:43 smithi090 ceph-mon[35142]: pgmap v828: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 165 op/s 2024-09-21T01:48:44.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 224439 30.88 MB/sec execute 271 sec latency 483.837 ms 2024-09-21T01:48:44.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48763 6.58 MB/sec execute 273 sec latency 165.102 ms 2024-09-21T01:48:45.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 224828 30.87 MB/sec execute 272 sec latency 517.351 ms 2024-09-21T01:48:45.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48860 6.58 MB/sec execute 274 sec latency 295.392 ms 2024-09-21T01:48:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:45 smithi045 ceph-mon[28361]: pgmap v829: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T01:48:46.230 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:45 smithi090 ceph-mon[35142]: pgmap v829: 129 pgs: 129 active+clean; 429 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.6 MiB/s wr, 132 op/s 2024-09-21T01:48:46.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 225086 30.79 MB/sec execute 273 sec latency 450.784 ms 2024-09-21T01:48:46.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 48974 6.58 MB/sec execute 275 sec latency 181.007 ms 2024-09-21T01:48:47.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 225450 30.73 MB/sec execute 274 sec latency 560.149 ms 2024-09-21T01:48:47.946 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49087 6.60 MB/sec execute 276 sec latency 150.754 ms 2024-09-21T01:48:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:47 smithi045 ceph-mon[28361]: pgmap v830: 129 pgs: 129 active+clean; 438 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.8 MiB/s wr, 175 op/s 2024-09-21T01:48:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:47 smithi090 ceph-mon[35142]: pgmap v830: 129 pgs: 129 active+clean; 438 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.8 MiB/s wr, 175 op/s 2024-09-21T01:48:48.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 225772 30.65 MB/sec execute 275 sec latency 701.859 ms 2024-09-21T01:48:48.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49215 6.63 MB/sec execute 277 sec latency 264.800 ms 2024-09-21T01:48:49.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 226036 30.61 MB/sec execute 276 sec latency 557.817 ms 2024-09-21T01:48:49.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49330 6.63 MB/sec execute 278 sec latency 187.724 ms 2024-09-21T01:48:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:49 smithi045 ceph-mon[28361]: pgmap v831: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.8 MiB/s wr, 157 op/s 2024-09-21T01:48:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:49 smithi090 ceph-mon[35142]: pgmap v831: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.8 MiB/s wr, 157 op/s 2024-09-21T01:48:50.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 226296 30.58 MB/sec execute 277 sec latency 396.244 ms 2024-09-21T01:48:50.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49425 6.63 MB/sec execute 279 sec latency 190.060 ms 2024-09-21T01:48:51.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 226628 30.55 MB/sec execute 278 sec latency 372.726 ms 2024-09-21T01:48:51.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49538 6.63 MB/sec execute 280 sec latency 162.679 ms 2024-09-21T01:48:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:51 smithi090 ceph-mon[35142]: pgmap v832: 129 pgs: 129 active+clean; 446 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.0 MiB/s wr, 129 op/s 2024-09-21T01:48:52.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:51 smithi045 ceph-mon[28361]: pgmap v832: 129 pgs: 129 active+clean; 446 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.0 MiB/s wr, 129 op/s 2024-09-21T01:48:52.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 227042 30.52 MB/sec execute 279 sec latency 484.024 ms 2024-09-21T01:48:52.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49611 6.64 MB/sec execute 281 sec latency 185.024 ms 2024-09-21T01:48:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:53.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:48:53.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 227467 30.50 MB/sec execute 280 sec latency 653.764 ms 2024-09-21T01:48:53.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49687 6.63 MB/sec execute 282 sec latency 140.577 ms 2024-09-21T01:48:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:53 smithi090 ceph-mon[35142]: pgmap v833: 129 pgs: 129 active+clean; 453 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 143 op/s 2024-09-21T01:48:54.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:53 smithi045 ceph-mon[28361]: pgmap v833: 129 pgs: 129 active+clean; 453 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 143 op/s 2024-09-21T01:48:54.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 227842 30.45 MB/sec execute 281 sec latency 159.379 ms 2024-09-21T01:48:54.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49781 6.62 MB/sec execute 283 sec latency 96.608 ms 2024-09-21T01:48:55.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 228410 30.45 MB/sec execute 282 sec latency 108.627 ms 2024-09-21T01:48:55.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 49905 6.63 MB/sec execute 284 sec latency 117.052 ms 2024-09-21T01:48:56.231 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:55 smithi090 ceph-mon[35142]: pgmap v834: 129 pgs: 129 active+clean; 453 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.2 MiB/s wr, 96 op/s 2024-09-21T01:48:56.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:55 smithi045 ceph-mon[28361]: pgmap v834: 129 pgs: 129 active+clean; 453 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.2 MiB/s wr, 96 op/s 2024-09-21T01:48:56.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 229858 30.61 MB/sec execute 283 sec latency 101.426 ms 2024-09-21T01:48:56.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50058 6.65 MB/sec execute 285 sec latency 114.852 ms 2024-09-21T01:48:57.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231019 30.74 MB/sec execute 284 sec latency 167.803 ms 2024-09-21T01:48:57.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50171 6.63 MB/sec execute 286 sec latency 140.311 ms 2024-09-21T01:48:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:57 smithi090 ceph-mon[35142]: pgmap v835: 129 pgs: 129 active+clean; 457 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 4.0 MiB/s wr, 117 op/s 2024-09-21T01:48:58.261 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:57 smithi045 ceph-mon[28361]: pgmap v835: 129 pgs: 129 active+clean; 457 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 4.0 MiB/s wr, 117 op/s 2024-09-21T01:48:58.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231326 30.69 MB/sec execute 285 sec latency 304.542 ms 2024-09-21T01:48:58.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50233 6.62 MB/sec execute 287 sec latency 358.796 ms 2024-09-21T01:48:59.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231341 30.58 MB/sec execute 286 sec latency 447.982 ms 2024-09-21T01:48:59.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50312 6.60 MB/sec execute 288 sec latency 174.101 ms 2024-09-21T01:49:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:48:59 smithi090 ceph-mon[35142]: pgmap v836: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.4 MiB/s wr, 95 op/s 2024-09-21T01:49:00.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:48:59 smithi045 ceph-mon[28361]: pgmap v836: 129 pgs: 129 active+clean; 460 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.4 MiB/s wr, 95 op/s 2024-09-21T01:49:00.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231382 30.48 MB/sec execute 287 sec latency 408.791 ms 2024-09-21T01:49:00.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50374 6.58 MB/sec execute 289 sec latency 307.845 ms 2024-09-21T01:49:01.707 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231405 30.38 MB/sec execute 288 sec latency 442.291 ms 2024-09-21T01:49:01.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50453 6.56 MB/sec execute 290 sec latency 224.158 ms 2024-09-21T01:49:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:01 smithi090 ceph-mon[35142]: pgmap v837: 129 pgs: 129 active+clean; 456 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.5 MiB/s wr, 141 op/s 2024-09-21T01:49:02.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:01 smithi045 ceph-mon[28361]: pgmap v837: 129 pgs: 129 active+clean; 456 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.5 MiB/s wr, 141 op/s 2024-09-21T01:49:02.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231451 30.28 MB/sec execute 289 sec latency 423.268 ms 2024-09-21T01:49:02.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50534 6.56 MB/sec execute 291 sec latency 244.694 ms 2024-09-21T01:49:03.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 231526 30.18 MB/sec execute 290 sec latency 434.130 ms 2024-09-21T01:49:03.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50638 6.54 MB/sec execute 292 sec latency 191.851 ms 2024-09-21T01:49:04.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:03 smithi090 ceph-mon[35142]: pgmap v838: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 222 op/s 2024-09-21T01:49:04.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:03 smithi045 ceph-mon[28361]: pgmap v838: 129 pgs: 129 active+clean; 453 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.3 MiB/s wr, 222 op/s 2024-09-21T01:49:04.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 232962 30.31 MB/sec execute 291 sec latency 198.641 ms 2024-09-21T01:49:04.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50774 6.54 MB/sec execute 293 sec latency 122.146 ms 2024-09-21T01:49:05.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 234449 30.53 MB/sec execute 292 sec latency 131.282 ms 2024-09-21T01:49:05.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 50924 6.55 MB/sec execute 294 sec latency 106.443 ms 2024-09-21T01:49:06.234 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:05 smithi090 ceph-mon[35142]: pgmap v839: 129 pgs: 129 active+clean; 454 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.6 MiB/s wr, 225 op/s 2024-09-21T01:49:06.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:05 smithi045 ceph-mon[28361]: pgmap v839: 129 pgs: 129 active+clean; 454 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.6 MiB/s wr, 225 op/s 2024-09-21T01:49:06.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 234923 30.51 MB/sec execute 293 sec latency 109.366 ms 2024-09-21T01:49:06.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51076 6.55 MB/sec execute 295 sec latency 108.838 ms 2024-09-21T01:49:07.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 235073 30.42 MB/sec execute 294 sec latency 187.535 ms 2024-09-21T01:49:07.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51215 6.55 MB/sec execute 296 sec latency 139.239 ms 2024-09-21T01:49:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:07 smithi090 ceph-mon[35142]: pgmap v840: 129 pgs: 129 active+clean; 439 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.2 MiB/s wr, 275 op/s 2024-09-21T01:49:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:08.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:07 smithi045 ceph-mon[28361]: pgmap v840: 129 pgs: 129 active+clean; 439 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.2 MiB/s wr, 275 op/s 2024-09-21T01:49:08.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:08.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 236317 30.50 MB/sec execute 295 sec latency 290.788 ms 2024-09-21T01:49:08.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51393 6.57 MB/sec execute 297 sec latency 114.802 ms 2024-09-21T01:49:09.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 236883 30.54 MB/sec execute 296 sec latency 379.394 ms 2024-09-21T01:49:09.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51503 6.55 MB/sec execute 298 sec latency 160.100 ms 2024-09-21T01:49:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:09 smithi090 ceph-mon[35142]: pgmap v841: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2024-09-21T01:49:10.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:09 smithi045 ceph-mon[28361]: pgmap v841: 129 pgs: 129 active+clean; 434 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.5 MiB/s wr, 266 op/s 2024-09-21T01:49:10.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 237128 30.50 MB/sec execute 297 sec latency 411.651 ms 2024-09-21T01:49:10.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51612 6.54 MB/sec execute 299 sec latency 168.393 ms 2024-09-21T01:49:11.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 237358 30.45 MB/sec execute 298 sec latency 548.924 ms 2024-09-21T01:49:11.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51758 6.53 MB/sec execute 300 sec latency 155.351 ms 2024-09-21T01:49:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:11 smithi090 ceph-mon[35142]: pgmap v842: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.3 MiB/s wr, 266 op/s 2024-09-21T01:49:12.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:11 smithi045 ceph-mon[28361]: pgmap v842: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.3 MiB/s wr, 266 op/s 2024-09-21T01:49:12.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 237697 30.40 MB/sec execute 299 sec latency 557.311 ms 2024-09-21T01:49:12.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 51883 6.52 MB/sec execute 301 sec latency 186.302 ms 2024-09-21T01:49:13.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 238105 30.38 MB/sec execute 300 sec latency 606.753 ms 2024-09-21T01:49:13.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52008 6.51 MB/sec execute 302 sec latency 172.020 ms 2024-09-21T01:49:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:13 smithi045 ceph-mon[28361]: pgmap v843: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.3 MiB/s wr, 247 op/s 2024-09-21T01:49:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:13 smithi090 ceph-mon[35142]: pgmap v843: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.3 MiB/s wr, 247 op/s 2024-09-21T01:49:14.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 238474 30.33 MB/sec execute 301 sec latency 452.949 ms 2024-09-21T01:49:14.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52177 6.53 MB/sec execute 303 sec latency 115.773 ms 2024-09-21T01:49:15.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 239194 30.35 MB/sec execute 302 sec latency 93.602 ms 2024-09-21T01:49:15.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52346 6.55 MB/sec execute 304 sec latency 87.895 ms 2024-09-21T01:49:16.233 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:15 smithi090 ceph-mon[35142]: pgmap v844: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.9 MiB/s wr, 165 op/s 2024-09-21T01:49:16.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:15 smithi045 ceph-mon[28361]: pgmap v844: 129 pgs: 129 active+clean; 420 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.9 MiB/s wr, 165 op/s 2024-09-21T01:49:16.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 240578 30.49 MB/sec execute 303 sec latency 117.097 ms 2024-09-21T01:49:16.947 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52487 6.56 MB/sec execute 305 sec latency 112.073 ms 2024-09-21T01:49:17.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 241848 30.64 MB/sec execute 304 sec latency 108.061 ms 2024-09-21T01:49:17.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52639 6.58 MB/sec execute 306 sec latency 119.544 ms 2024-09-21T01:49:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:17 smithi090 ceph-mon[35142]: pgmap v845: 129 pgs: 129 active+clean; 426 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-09-21T01:49:18.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:17 smithi045 ceph-mon[28361]: pgmap v845: 129 pgs: 129 active+clean; 426 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.7 MiB/s wr, 177 op/s 2024-09-21T01:49:18.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 242091 30.57 MB/sec execute 305 sec latency 88.834 ms 2024-09-21T01:49:18.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52830 6.62 MB/sec execute 307 sec latency 87.815 ms 2024-09-21T01:49:19.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 243409 30.66 MB/sec execute 306 sec latency 94.099 ms 2024-09-21T01:49:19.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 52985 6.64 MB/sec execute 308 sec latency 115.673 ms 2024-09-21T01:49:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:19 smithi090 ceph-mon[35142]: pgmap v846: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-09-21T01:49:20.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:19 smithi045 ceph-mon[28361]: pgmap v846: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-09-21T01:49:20.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 244535 30.82 MB/sec execute 307 sec latency 125.512 ms 2024-09-21T01:49:20.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53102 6.63 MB/sec execute 309 sec latency 126.000 ms 2024-09-21T01:49:21.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 245539 30.88 MB/sec execute 308 sec latency 101.309 ms 2024-09-21T01:49:21.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53202 6.64 MB/sec execute 310 sec latency 110.223 ms 2024-09-21T01:49:22.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:21 smithi090 ceph-mon[35142]: pgmap v847: 129 pgs: 129 active+clean; 437 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.2 MiB/s wr, 145 op/s 2024-09-21T01:49:22.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:21 smithi045 ceph-mon[28361]: pgmap v847: 129 pgs: 129 active+clean; 437 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.2 MiB/s wr, 145 op/s 2024-09-21T01:49:22.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 246096 30.88 MB/sec execute 309 sec latency 104.275 ms 2024-09-21T01:49:22.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53291 6.63 MB/sec execute 311 sec latency 134.675 ms 2024-09-21T01:49:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:23.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:23.708 INFO:tasks.workunit.client.0.smithi045.stdout: 10 247575 31.02 MB/sec execute 310 sec latency 111.274 ms 2024-09-21T01:49:23.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53407 6.64 MB/sec execute 312 sec latency 94.265 ms 2024-09-21T01:49:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:23 smithi090 ceph-mon[35142]: pgmap v848: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 135 op/s 2024-09-21T01:49:24.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:23 smithi045 ceph-mon[28361]: pgmap v848: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 135 op/s 2024-09-21T01:49:24.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 248924 31.18 MB/sec execute 311 sec latency 105.870 ms 2024-09-21T01:49:24.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53556 6.66 MB/sec execute 313 sec latency 121.357 ms 2024-09-21T01:49:25.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:24 smithi090 ceph-mon[35142]: pgmap v849: 129 pgs: 129 active+clean; 448 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 122 op/s 2024-09-21T01:49:25.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:24 smithi045 ceph-mon[28361]: pgmap v849: 129 pgs: 129 active+clean; 448 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 122 op/s 2024-09-21T01:49:25.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 249117 31.10 MB/sec execute 312 sec latency 250.940 ms 2024-09-21T01:49:25.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53676 6.64 MB/sec execute 314 sec latency 133.393 ms 2024-09-21T01:49:26.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 249918 31.12 MB/sec execute 313 sec latency 100.894 ms 2024-09-21T01:49:26.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53789 6.64 MB/sec execute 315 sec latency 148.094 ms 2024-09-21T01:49:27.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 251374 31.29 MB/sec execute 314 sec latency 106.876 ms 2024-09-21T01:49:27.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:27 smithi090 ceph-mon[35142]: pgmap v850: 129 pgs: 129 active+clean; 447 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.5 MiB/s wr, 131 op/s 2024-09-21T01:49:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:27 smithi045 ceph-mon[28361]: pgmap v850: 129 pgs: 129 active+clean; 447 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.5 MiB/s wr, 131 op/s 2024-09-21T01:49:27.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 53903 6.62 MB/sec execute 316 sec latency 171.073 ms 2024-09-21T01:49:28.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 252594 31.41 MB/sec execute 315 sec latency 150.569 ms 2024-09-21T01:49:28.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54016 6.62 MB/sec execute 317 sec latency 236.734 ms 2024-09-21T01:49:29.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 252680 31.32 MB/sec execute 316 sec latency 291.253 ms 2024-09-21T01:49:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:29 smithi090 ceph-mon[35142]: pgmap v851: 129 pgs: 129 active+clean; 451 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.3 MiB/s wr, 123 op/s 2024-09-21T01:49:29.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:29 smithi045 ceph-mon[28361]: pgmap v851: 129 pgs: 129 active+clean; 451 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.3 MiB/s wr, 123 op/s 2024-09-21T01:49:29.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54112 6.61 MB/sec execute 318 sec latency 233.037 ms 2024-09-21T01:49:30.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 253240 31.31 MB/sec execute 317 sec latency 359.614 ms 2024-09-21T01:49:30.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54231 6.60 MB/sec execute 319 sec latency 222.258 ms 2024-09-21T01:49:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:31 smithi090 ceph-mon[35142]: pgmap v852: 129 pgs: 129 active+clean; 449 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.1 MiB/s wr, 122 op/s 2024-09-21T01:49:31.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 254666 31.44 MB/sec execute 318 sec latency 102.301 ms 2024-09-21T01:49:31.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:31 smithi045 ceph-mon[28361]: pgmap v852: 129 pgs: 129 active+clean; 449 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 4.1 MiB/s wr, 122 op/s 2024-09-21T01:49:31.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54363 6.59 MB/sec execute 320 sec latency 149.797 ms 2024-09-21T01:49:32.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 256133 31.61 MB/sec execute 319 sec latency 101.448 ms 2024-09-21T01:49:32.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54515 6.60 MB/sec execute 321 sec latency 130.314 ms 2024-09-21T01:49:33.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 256273 31.53 MB/sec execute 320 sec latency 100.465 ms 2024-09-21T01:49:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:33 smithi090 ceph-mon[35142]: pgmap v853: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 139 op/s 2024-09-21T01:49:33.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:33 smithi045 ceph-mon[28361]: pgmap v853: 129 pgs: 129 active+clean; 444 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.8 MiB/s wr, 139 op/s 2024-09-21T01:49:33.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54662 6.60 MB/sec execute 322 sec latency 113.492 ms 2024-09-21T01:49:34.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 257116 31.56 MB/sec execute 321 sec latency 211.847 ms 2024-09-21T01:49:34.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54829 6.60 MB/sec execute 323 sec latency 104.052 ms 2024-09-21T01:49:35.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 257782 31.55 MB/sec execute 322 sec latency 388.126 ms 2024-09-21T01:49:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:35 smithi090 ceph-mon[35142]: pgmap v854: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-09-21T01:49:35.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:35 smithi045 ceph-mon[28361]: pgmap v854: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.9 MiB/s wr, 131 op/s 2024-09-21T01:49:35.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 54988 6.61 MB/sec execute 324 sec latency 115.358 ms 2024-09-21T01:49:36.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 258058 31.52 MB/sec execute 323 sec latency 422.999 ms 2024-09-21T01:49:36.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55097 6.60 MB/sec execute 325 sec latency 200.021 ms 2024-09-21T01:49:37.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 258402 31.50 MB/sec execute 324 sec latency 468.242 ms 2024-09-21T01:49:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:37 smithi090 ceph-mon[35142]: pgmap v855: 129 pgs: 129 active+clean; 435 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.5 MiB/s wr, 173 op/s 2024-09-21T01:49:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:49:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:37 smithi045 ceph-mon[28361]: pgmap v855: 129 pgs: 129 active+clean; 435 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.5 MiB/s wr, 173 op/s 2024-09-21T01:49:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:49:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:37.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55196 6.58 MB/sec execute 326 sec latency 190.298 ms 2024-09-21T01:49:38.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 258605 31.44 MB/sec execute 325 sec latency 504.167 ms 2024-09-21T01:49:38.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55317 6.57 MB/sec execute 327 sec latency 194.608 ms 2024-09-21T01:49:39.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 258959 31.40 MB/sec execute 326 sec latency 565.918 ms 2024-09-21T01:49:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:39 smithi090 ceph-mon[35142]: pgmap v856: 129 pgs: 129 active+clean; 434 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-09-21T01:49:39.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:39 smithi045 ceph-mon[28361]: pgmap v856: 129 pgs: 129 active+clean; 434 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 2.8 MiB/s wr, 174 op/s 2024-09-21T01:49:39.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55430 6.57 MB/sec execute 328 sec latency 196.667 ms 2024-09-21T01:49:40.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259366 31.37 MB/sec execute 327 sec latency 537.356 ms 2024-09-21T01:49:40.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55538 6.56 MB/sec execute 329 sec latency 172.030 ms 2024-09-21T01:49:40.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:40 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:49:40.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:40 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:49:41.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:40 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:49:41.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:40 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:49:41.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259563 31.31 MB/sec execute 328 sec latency 420.210 ms 2024-09-21T01:49:41.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55699 6.57 MB/sec execute 330 sec latency 154.846 ms 2024-09-21T01:49:41.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:41 smithi090 ceph-mon[35142]: pgmap v857: 129 pgs: 129 active+clean; 436 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-09-21T01:49:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:41 smithi045 ceph-mon[28361]: pgmap v857: 129 pgs: 129 active+clean; 436 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.4 MiB/s wr, 161 op/s 2024-09-21T01:49:42.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259688 31.24 MB/sec execute 329 sec latency 351.038 ms 2024-09-21T01:49:42.948 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55793 6.57 MB/sec execute 331 sec latency 144.459 ms 2024-09-21T01:49:43.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259722 31.14 MB/sec execute 330 sec latency 342.283 ms 2024-09-21T01:49:43.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 55936 6.58 MB/sec execute 332 sec latency 150.227 ms 2024-09-21T01:49:43.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:43 smithi090 ceph-mon[35142]: pgmap v858: 129 pgs: 129 active+clean; 437 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-09-21T01:49:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:43 smithi045 ceph-mon[28361]: pgmap v858: 129 pgs: 129 active+clean; 437 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.4 MiB/s wr, 173 op/s 2024-09-21T01:49:44.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259755 31.05 MB/sec execute 331 sec latency 336.598 ms 2024-09-21T01:49:44.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56058 6.58 MB/sec execute 333 sec latency 179.384 ms 2024-09-21T01:49:45.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259775 30.96 MB/sec execute 332 sec latency 345.809 ms 2024-09-21T01:49:45.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56170 6.60 MB/sec execute 334 sec latency 146.676 ms 2024-09-21T01:49:45.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:45 smithi090 ceph-mon[35142]: pgmap v859: 129 pgs: 129 active+clean; 439 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 138 op/s 2024-09-21T01:49:46.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:45 smithi045 ceph-mon[28361]: pgmap v859: 129 pgs: 129 active+clean; 439 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 138 op/s 2024-09-21T01:49:46.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259854 30.87 MB/sec execute 333 sec latency 350.643 ms 2024-09-21T01:49:46.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56291 6.61 MB/sec execute 335 sec latency 182.461 ms 2024-09-21T01:49:47.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 259914 30.78 MB/sec execute 334 sec latency 342.524 ms 2024-09-21T01:49:47.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56384 6.62 MB/sec execute 336 sec latency 195.840 ms 2024-09-21T01:49:47.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:47 smithi090 ceph-mon[35142]: pgmap v860: 129 pgs: 129 active+clean; 429 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.2 MiB/s wr, 169 op/s 2024-09-21T01:49:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:47 smithi045 ceph-mon[28361]: pgmap v860: 129 pgs: 129 active+clean; 429 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.2 MiB/s wr, 169 op/s 2024-09-21T01:49:48.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 260325 30.77 MB/sec execute 335 sec latency 461.484 ms 2024-09-21T01:49:48.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56502 6.63 MB/sec execute 337 sec latency 207.742 ms 2024-09-21T01:49:49.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 260558 30.70 MB/sec execute 336 sec latency 520.420 ms 2024-09-21T01:49:49.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56576 6.62 MB/sec execute 338 sec latency 194.199 ms 2024-09-21T01:49:49.997 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:49 smithi045 ceph-mon[28361]: pgmap v861: 129 pgs: 129 active+clean; 428 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.7 MiB/s wr, 111 op/s 2024-09-21T01:49:49.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:49 smithi090 ceph-mon[35142]: pgmap v861: 129 pgs: 129 active+clean; 428 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.7 MiB/s wr, 111 op/s 2024-09-21T01:49:50.709 INFO:tasks.workunit.client.0.smithi045.stdout: 10 260721 30.63 MB/sec execute 337 sec latency 610.678 ms 2024-09-21T01:49:50.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56696 6.63 MB/sec execute 339 sec latency 199.857 ms 2024-09-21T01:49:51.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 261195 30.59 MB/sec execute 338 sec latency 420.802 ms 2024-09-21T01:49:51.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56745 6.61 MB/sec execute 340 sec latency 180.675 ms 2024-09-21T01:49:51.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:51 smithi090 ceph-mon[35142]: pgmap v862: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.9 MiB/s wr, 109 op/s 2024-09-21T01:49:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:51 smithi045 ceph-mon[28361]: pgmap v862: 129 pgs: 129 active+clean; 431 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.9 MiB/s wr, 109 op/s 2024-09-21T01:49:52.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 261438 30.57 MB/sec execute 339 sec latency 391.584 ms 2024-09-21T01:49:52.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56807 6.61 MB/sec execute 341 sec latency 174.005 ms 2024-09-21T01:49:52.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:49:53.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 262046 30.59 MB/sec execute 340 sec latency 413.280 ms 2024-09-21T01:49:53.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 56894 6.60 MB/sec execute 342 sec latency 157.757 ms 2024-09-21T01:49:53.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:53 smithi090 ceph-mon[35142]: pgmap v863: 129 pgs: 129 active+clean; 437 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 110 op/s 2024-09-21T01:49:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:53 smithi045 ceph-mon[28361]: pgmap v863: 129 pgs: 129 active+clean; 437 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 110 op/s 2024-09-21T01:49:54.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 263225 30.70 MB/sec execute 341 sec latency 125.194 ms 2024-09-21T01:49:54.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57020 6.61 MB/sec execute 343 sec latency 125.629 ms 2024-09-21T01:49:55.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 263365 30.62 MB/sec execute 342 sec latency 114.548 ms 2024-09-21T01:49:55.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57150 6.61 MB/sec execute 344 sec latency 137.091 ms 2024-09-21T01:49:55.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:55 smithi090 ceph-mon[35142]: pgmap v864: 129 pgs: 129 active+clean; 438 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.1 MiB/s wr, 87 op/s 2024-09-21T01:49:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:55 smithi045 ceph-mon[28361]: pgmap v864: 129 pgs: 129 active+clean; 438 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.1 MiB/s wr, 87 op/s 2024-09-21T01:49:56.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 264208 30.65 MB/sec execute 343 sec latency 178.812 ms 2024-09-21T01:49:56.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57265 6.61 MB/sec execute 345 sec latency 147.066 ms 2024-09-21T01:49:57.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 265365 30.74 MB/sec execute 344 sec latency 215.632 ms 2024-09-21T01:49:57.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57379 6.60 MB/sec execute 346 sec latency 111.812 ms 2024-09-21T01:49:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:57 smithi045 ceph-mon[28361]: pgmap v865: 129 pgs: 129 active+clean; 441 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.0 MiB/s wr, 118 op/s 2024-09-21T01:49:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:57 smithi090 ceph-mon[35142]: pgmap v865: 129 pgs: 129 active+clean; 441 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.0 MiB/s wr, 118 op/s 2024-09-21T01:49:58.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266468 30.85 MB/sec execute 345 sec latency 218.520 ms 2024-09-21T01:49:58.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57467 6.58 MB/sec execute 347 sec latency 220.706 ms 2024-09-21T01:49:59.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266655 30.79 MB/sec execute 346 sec latency 470.218 ms 2024-09-21T01:49:59.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57555 6.58 MB/sec execute 348 sec latency 215.632 ms 2024-09-21T01:50:00.090 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:49:59 smithi045 ceph-mon[28361]: pgmap v866: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.0 MiB/s wr, 88 op/s 2024-09-21T01:50:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:49:59 smithi090 ceph-mon[35142]: pgmap v866: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.0 MiB/s wr, 88 op/s 2024-09-21T01:50:00.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266778 30.72 MB/sec execute 347 sec latency 517.778 ms 2024-09-21T01:50:00.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57636 6.57 MB/sec execute 349 sec latency 194.604 ms 2024-09-21T01:50:01.149 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:00 smithi090 ceph-mon[35142]: overall HEALTH_OK 2024-09-21T01:50:01.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:00 smithi045 ceph-mon[28361]: overall HEALTH_OK 2024-09-21T01:50:01.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266785 30.63 MB/sec execute 348 sec latency 340.838 ms 2024-09-21T01:50:01.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57748 6.56 MB/sec execute 350 sec latency 177.643 ms 2024-09-21T01:50:02.038 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:01 smithi045 ceph-mon[28361]: pgmap v867: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.0 MiB/s wr, 90 op/s 2024-09-21T01:50:02.163 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:01 smithi090 ceph-mon[35142]: pgmap v867: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.0 MiB/s wr, 90 op/s 2024-09-21T01:50:02.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266837 30.54 MB/sec execute 349 sec latency 449.878 ms 2024-09-21T01:50:02.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57852 6.55 MB/sec execute 351 sec latency 207.010 ms 2024-09-21T01:50:03.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266849 30.46 MB/sec execute 350 sec latency 419.848 ms 2024-09-21T01:50:03.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 57988 6.56 MB/sec execute 352 sec latency 147.355 ms 2024-09-21T01:50:03.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:03 smithi090 ceph-mon[35142]: pgmap v868: 129 pgs: 129 active+clean; 434 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 MiB/s wr, 117 op/s 2024-09-21T01:50:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:03 smithi045 ceph-mon[28361]: pgmap v868: 129 pgs: 129 active+clean; 434 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 MiB/s wr, 117 op/s 2024-09-21T01:50:04.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266912 30.38 MB/sec execute 351 sec latency 436.189 ms 2024-09-21T01:50:04.949 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58112 6.55 MB/sec execute 353 sec latency 165.436 ms 2024-09-21T01:50:05.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 266945 30.29 MB/sec execute 352 sec latency 623.063 ms 2024-09-21T01:50:05.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58231 6.54 MB/sec execute 354 sec latency 149.712 ms 2024-09-21T01:50:05.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:05 smithi090 ceph-mon[35142]: pgmap v869: 129 pgs: 129 active+clean; 433 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.6 MiB/s wr, 105 op/s 2024-09-21T01:50:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:05 smithi045 ceph-mon[28361]: pgmap v869: 129 pgs: 129 active+clean; 433 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.6 MiB/s wr, 105 op/s 2024-09-21T01:50:06.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 267317 30.28 MB/sec execute 353 sec latency 317.737 ms 2024-09-21T01:50:06.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58396 6.55 MB/sec execute 355 sec latency 111.121 ms 2024-09-21T01:50:07.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 267648 30.21 MB/sec execute 354 sec latency 329.945 ms 2024-09-21T01:50:07.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58545 6.56 MB/sec execute 356 sec latency 117.037 ms 2024-09-21T01:50:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:07 smithi045 ceph-mon[28361]: pgmap v870: 129 pgs: 129 active+clean; 427 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.0 MiB/s wr, 143 op/s 2024-09-21T01:50:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:07 smithi090 ceph-mon[35142]: pgmap v870: 129 pgs: 129 active+clean; 427 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.0 MiB/s wr, 143 op/s 2024-09-21T01:50:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:08.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 268133 30.19 MB/sec execute 355 sec latency 326.550 ms 2024-09-21T01:50:08.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58664 6.55 MB/sec execute 357 sec latency 145.498 ms 2024-09-21T01:50:09.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 268385 30.14 MB/sec execute 356 sec latency 410.237 ms 2024-09-21T01:50:09.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58785 6.53 MB/sec execute 358 sec latency 131.722 ms 2024-09-21T01:50:10.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:09 smithi045 ceph-mon[28361]: pgmap v871: 129 pgs: 129 active+clean; 425 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-09-21T01:50:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:09 smithi090 ceph-mon[35142]: pgmap v871: 129 pgs: 129 active+clean; 425 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.4 MiB/s wr, 151 op/s 2024-09-21T01:50:10.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 268831 30.15 MB/sec execute 357 sec latency 401.727 ms 2024-09-21T01:50:10.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 58931 6.53 MB/sec execute 359 sec latency 173.126 ms 2024-09-21T01:50:11.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 269205 30.14 MB/sec execute 358 sec latency 388.500 ms 2024-09-21T01:50:11.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59062 6.53 MB/sec execute 360 sec latency 128.462 ms 2024-09-21T01:50:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:11 smithi045 ceph-mon[28361]: pgmap v872: 129 pgs: 129 active+clean; 425 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-09-21T01:50:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:11 smithi090 ceph-mon[35142]: pgmap v872: 129 pgs: 129 active+clean; 425 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 2.3 MiB/s wr, 157 op/s 2024-09-21T01:50:12.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 269863 30.15 MB/sec execute 359 sec latency 350.271 ms 2024-09-21T01:50:12.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59236 6.53 MB/sec execute 361 sec latency 129.895 ms 2024-09-21T01:50:13.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270018 30.10 MB/sec execute 360 sec latency 351.130 ms 2024-09-21T01:50:13.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59371 6.54 MB/sec execute 362 sec latency 125.080 ms 2024-09-21T01:50:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:13 smithi045 ceph-mon[28361]: pgmap v873: 129 pgs: 129 active+clean; 426 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-09-21T01:50:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:13 smithi090 ceph-mon[35142]: pgmap v873: 129 pgs: 129 active+clean; 426 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.3 MiB/s wr, 169 op/s 2024-09-21T01:50:14.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270320 30.06 MB/sec execute 361 sec latency 300.371 ms 2024-09-21T01:50:14.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59482 6.55 MB/sec execute 363 sec latency 180.062 ms 2024-09-21T01:50:15.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:50:15.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:50:15.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:15 smithi045 ceph-mon[28361]: pgmap v874: 129 pgs: 129 active+clean; 428 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.8 MiB/s wr, 145 op/s 2024-09-21T01:50:15.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:50:15.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:50:15.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:50:15.710 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270329 29.97 MB/sec execute 362 sec latency 343.619 ms 2024-09-21T01:50:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:50:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:50:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:15 smithi090 ceph-mon[35142]: pgmap v874: 129 pgs: 129 active+clean; 428 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.8 MiB/s wr, 145 op/s 2024-09-21T01:50:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:50:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:50:15.749 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:50:15.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59562 6.54 MB/sec execute 364 sec latency 234.960 ms 2024-09-21T01:50:16.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270387 29.89 MB/sec execute 363 sec latency 402.192 ms 2024-09-21T01:50:16.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59676 6.55 MB/sec execute 365 sec latency 138.678 ms 2024-09-21T01:50:17.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:17 smithi045 ceph-mon[28361]: pgmap v875: 129 pgs: 129 active+clean; 437 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-09-21T01:50:17.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270412 29.81 MB/sec execute 364 sec latency 267.217 ms 2024-09-21T01:50:17.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:17 smithi090 ceph-mon[35142]: pgmap v875: 129 pgs: 129 active+clean; 437 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-09-21T01:50:17.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59802 6.56 MB/sec execute 366 sec latency 159.954 ms 2024-09-21T01:50:18.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270489 29.74 MB/sec execute 365 sec latency 309.333 ms 2024-09-21T01:50:18.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 59912 6.58 MB/sec execute 367 sec latency 172.682 ms 2024-09-21T01:50:19.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:19 smithi045 ceph-mon[28361]: pgmap v876: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-09-21T01:50:19.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 270859 29.73 MB/sec execute 366 sec latency 324.196 ms 2024-09-21T01:50:19.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:19 smithi090 ceph-mon[35142]: pgmap v876: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.8 MiB/s wr, 142 op/s 2024-09-21T01:50:19.950 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60057 6.59 MB/sec execute 368 sec latency 142.352 ms 2024-09-21T01:50:20.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 271198 29.67 MB/sec execute 367 sec latency 353.016 ms 2024-09-21T01:50:20.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60186 6.59 MB/sec execute 369 sec latency 112.117 ms 2024-09-21T01:50:21.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:21 smithi090 ceph-mon[35142]: pgmap v877: 129 pgs: 129 active+clean; 443 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-09-21T01:50:21.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:21 smithi045 ceph-mon[28361]: pgmap v877: 129 pgs: 129 active+clean; 443 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.8 MiB/s wr, 120 op/s 2024-09-21T01:50:21.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 271698 29.65 MB/sec execute 368 sec latency 363.432 ms 2024-09-21T01:50:21.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60273 6.59 MB/sec execute 370 sec latency 148.452 ms 2024-09-21T01:50:22.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:22.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 271986 29.61 MB/sec execute 369 sec latency 368.150 ms 2024-09-21T01:50:22.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:22.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60356 6.59 MB/sec execute 371 sec latency 159.915 ms 2024-09-21T01:50:23.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:23 smithi045 ceph-mon[28361]: pgmap v878: 129 pgs: 129 active+clean; 452 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.6 MiB/s wr, 119 op/s 2024-09-21T01:50:23.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 272414 29.61 MB/sec execute 370 sec latency 403.173 ms 2024-09-21T01:50:23.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:23 smithi090 ceph-mon[35142]: pgmap v878: 129 pgs: 129 active+clean; 452 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.6 MiB/s wr, 119 op/s 2024-09-21T01:50:23.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60426 6.58 MB/sec execute 372 sec latency 132.537 ms 2024-09-21T01:50:24.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 272784 29.61 MB/sec execute 371 sec latency 413.093 ms 2024-09-21T01:50:24.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60534 6.59 MB/sec execute 373 sec latency 163.838 ms 2024-09-21T01:50:25.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:25 smithi045 ceph-mon[28361]: pgmap v879: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.4 MiB/s wr, 97 op/s 2024-09-21T01:50:25.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 273371 29.61 MB/sec execute 372 sec latency 320.394 ms 2024-09-21T01:50:25.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:25 smithi090 ceph-mon[35142]: pgmap v879: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.4 MiB/s wr, 97 op/s 2024-09-21T01:50:25.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60654 6.59 MB/sec execute 374 sec latency 142.385 ms 2024-09-21T01:50:26.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 273873 29.60 MB/sec execute 373 sec latency 286.137 ms 2024-09-21T01:50:26.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60772 6.59 MB/sec execute 375 sec latency 128.551 ms 2024-09-21T01:50:27.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 274033 29.53 MB/sec execute 374 sec latency 116.892 ms 2024-09-21T01:50:27.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:27 smithi090 ceph-mon[35142]: pgmap v880: 129 pgs: 129 active+clean; 456 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 119 op/s 2024-09-21T01:50:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:27 smithi045 ceph-mon[28361]: pgmap v880: 129 pgs: 129 active+clean; 456 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 119 op/s 2024-09-21T01:50:27.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60890 6.58 MB/sec execute 376 sec latency 110.449 ms 2024-09-21T01:50:28.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 275058 29.59 MB/sec execute 375 sec latency 133.304 ms 2024-09-21T01:50:28.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 60996 6.57 MB/sec execute 377 sec latency 147.053 ms 2024-09-21T01:50:29.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 276314 29.72 MB/sec execute 376 sec latency 116.182 ms 2024-09-21T01:50:29.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:29 smithi090 ceph-mon[35142]: pgmap v881: 129 pgs: 129 active+clean; 455 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 88 op/s 2024-09-21T01:50:29.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:29 smithi045 ceph-mon[28361]: pgmap v881: 129 pgs: 129 active+clean; 455 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 88 op/s 2024-09-21T01:50:29.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61119 6.56 MB/sec execute 378 sec latency 107.793 ms 2024-09-21T01:50:30.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 277396 29.79 MB/sec execute 377 sec latency 155.598 ms 2024-09-21T01:50:30.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61221 6.55 MB/sec execute 379 sec latency 131.633 ms 2024-09-21T01:50:31.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:31 smithi090 ceph-mon[35142]: pgmap v882: 129 pgs: 129 active+clean; 451 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.2 MiB/s wr, 91 op/s 2024-09-21T01:50:31.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:31 smithi045 ceph-mon[28361]: pgmap v882: 129 pgs: 129 active+clean; 451 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.2 MiB/s wr, 91 op/s 2024-09-21T01:50:31.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 277432 29.72 MB/sec execute 378 sec latency 271.640 ms 2024-09-21T01:50:31.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61332 6.55 MB/sec execute 380 sec latency 175.446 ms 2024-09-21T01:50:32.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 277478 29.64 MB/sec execute 379 sec latency 356.224 ms 2024-09-21T01:50:32.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61460 6.54 MB/sec execute 381 sec latency 142.730 ms 2024-09-21T01:50:33.711 INFO:tasks.workunit.client.0.smithi045.stdout: 10 277531 29.57 MB/sec execute 380 sec latency 250.505 ms 2024-09-21T01:50:33.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:33 smithi090 ceph-mon[35142]: pgmap v883: 129 pgs: 129 active+clean; 447 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.6 MiB/s wr, 123 op/s 2024-09-21T01:50:33.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:33 smithi045 ceph-mon[28361]: pgmap v883: 129 pgs: 129 active+clean; 447 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 3.6 MiB/s wr, 123 op/s 2024-09-21T01:50:33.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61587 6.54 MB/sec execute 382 sec latency 134.377 ms 2024-09-21T01:50:34.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 277597 29.49 MB/sec execute 381 sec latency 288.159 ms 2024-09-21T01:50:34.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61694 6.54 MB/sec execute 383 sec latency 165.329 ms 2024-09-21T01:50:35.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 278087 29.50 MB/sec execute 382 sec latency 357.403 ms 2024-09-21T01:50:35.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:35 smithi090 ceph-mon[35142]: pgmap v884: 129 pgs: 129 active+clean; 446 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.6 MiB/s wr, 117 op/s 2024-09-21T01:50:35.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:35 smithi045 ceph-mon[28361]: pgmap v884: 129 pgs: 129 active+clean; 446 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.6 MiB/s wr, 117 op/s 2024-09-21T01:50:35.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61815 6.53 MB/sec execute 384 sec latency 126.010 ms 2024-09-21T01:50:36.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 278264 29.42 MB/sec execute 383 sec latency 470.240 ms 2024-09-21T01:50:36.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 61974 6.54 MB/sec execute 385 sec latency 125.514 ms 2024-09-21T01:50:37.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 278916 29.41 MB/sec execute 384 sec latency 422.604 ms 2024-09-21T01:50:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:37 smithi090 ceph-mon[35142]: pgmap v885: 129 pgs: 129 active+clean; 422 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.3 MiB/s wr, 172 op/s 2024-09-21T01:50:37.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:37 smithi045 ceph-mon[28361]: pgmap v885: 129 pgs: 129 active+clean; 422 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.3 MiB/s wr, 172 op/s 2024-09-21T01:50:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:37.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 62117 6.54 MB/sec execute 386 sec latency 115.406 ms 2024-09-21T01:50:38.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 279329 29.42 MB/sec execute 385 sec latency 359.350 ms 2024-09-21T01:50:38.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 62244 6.53 MB/sec execute 387 sec latency 165.889 ms 2024-09-21T01:50:39.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:39 smithi045 ceph-mon[28361]: pgmap v886: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.5 MiB/s wr, 145 op/s 2024-09-21T01:50:39.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 279708 29.42 MB/sec execute 386 sec latency 366.862 ms 2024-09-21T01:50:39.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:39 smithi090 ceph-mon[35142]: pgmap v886: 129 pgs: 129 active+clean; 416 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 2.5 MiB/s wr, 145 op/s 2024-09-21T01:50:39.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 62356 6.52 MB/sec execute 388 sec latency 263.985 ms 2024-09-21T01:50:40.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 280070 29.39 MB/sec execute 387 sec latency 380.853 ms 2024-09-21T01:50:40.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 62531 6.52 MB/sec execute 389 sec latency 123.449 ms 2024-09-21T01:50:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:41 smithi090 ceph-mon[35142]: pgmap v887: 129 pgs: 129 active+clean; 418 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 2.6 MiB/s wr, 175 op/s 2024-09-21T01:50:41.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 280751 29.41 MB/sec execute 388 sec latency 306.952 ms 2024-09-21T01:50:41.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:41 smithi045 ceph-mon[28361]: pgmap v887: 129 pgs: 129 active+clean; 418 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 2.6 MiB/s wr, 175 op/s 2024-09-21T01:50:41.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 62690 6.53 MB/sec execute 390 sec latency 103.132 ms 2024-09-21T01:50:42.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 281030 29.37 MB/sec execute 389 sec latency 258.412 ms 2024-09-21T01:50:42.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 62826 6.53 MB/sec execute 391 sec latency 152.589 ms 2024-09-21T01:50:43.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 281678 29.38 MB/sec execute 390 sec latency 132.730 ms 2024-09-21T01:50:43.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:43 smithi090 ceph-mon[35142]: pgmap v888: 129 pgs: 129 active+clean; 420 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 187 op/s 2024-09-21T01:50:43.880 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:43 smithi045 ceph-mon[28361]: pgmap v888: 129 pgs: 129 active+clean; 420 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 187 op/s 2024-09-21T01:50:43.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63009 6.55 MB/sec execute 392 sec latency 163.871 ms 2024-09-21T01:50:44.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 282952 29.47 MB/sec execute 391 sec latency 132.486 ms 2024-09-21T01:50:44.951 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63161 6.56 MB/sec execute 393 sec latency 90.682 ms 2024-09-21T01:50:45.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 284189 29.59 MB/sec execute 392 sec latency 158.316 ms 2024-09-21T01:50:45.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:45 smithi090 ceph-mon[35142]: pgmap v889: 129 pgs: 129 active+clean; 421 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-09-21T01:50:45.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:45 smithi045 ceph-mon[28361]: pgmap v889: 129 pgs: 129 active+clean; 421 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.0 MiB/s wr, 139 op/s 2024-09-21T01:50:45.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63315 6.58 MB/sec execute 394 sec latency 108.344 ms 2024-09-21T01:50:46.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 284506 29.55 MB/sec execute 393 sec latency 375.495 ms 2024-09-21T01:50:46.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63417 6.58 MB/sec execute 395 sec latency 217.627 ms 2024-09-21T01:50:47.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 284530 29.48 MB/sec execute 394 sec latency 447.966 ms 2024-09-21T01:50:47.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:47 smithi090 ceph-mon[35142]: pgmap v890: 129 pgs: 129 active+clean; 429 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.2 MiB/s wr, 176 op/s 2024-09-21T01:50:47.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:47 smithi045 ceph-mon[28361]: pgmap v890: 129 pgs: 129 active+clean; 429 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.2 MiB/s wr, 176 op/s 2024-09-21T01:50:47.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63552 6.59 MB/sec execute 396 sec latency 213.155 ms 2024-09-21T01:50:48.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 284568 29.41 MB/sec execute 395 sec latency 486.398 ms 2024-09-21T01:50:48.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63659 6.60 MB/sec execute 397 sec latency 177.749 ms 2024-09-21T01:50:49.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 284574 29.33 MB/sec execute 396 sec latency 467.098 ms 2024-09-21T01:50:49.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:49 smithi090 ceph-mon[35142]: pgmap v891: 129 pgs: 129 active+clean; 436 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.2 MiB/s wr, 130 op/s 2024-09-21T01:50:49.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:49 smithi045 ceph-mon[28361]: pgmap v891: 129 pgs: 129 active+clean; 436 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.2 MiB/s wr, 130 op/s 2024-09-21T01:50:49.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63738 6.59 MB/sec execute 398 sec latency 191.433 ms 2024-09-21T01:50:50.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 284640 29.26 MB/sec execute 397 sec latency 425.557 ms 2024-09-21T01:50:50.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63820 6.59 MB/sec execute 399 sec latency 286.754 ms 2024-09-21T01:50:51.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:51 smithi090 ceph-mon[35142]: pgmap v892: 129 pgs: 129 active+clean; 440 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-09-21T01:50:51.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 285057 29.26 MB/sec execute 398 sec latency 400.949 ms 2024-09-21T01:50:51.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:51 smithi045 ceph-mon[28361]: pgmap v892: 129 pgs: 129 active+clean; 440 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-09-21T01:50:51.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63899 6.59 MB/sec execute 400 sec latency 166.966 ms 2024-09-21T01:50:52.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 286276 29.33 MB/sec execute 399 sec latency 165.832 ms 2024-09-21T01:50:52.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:52.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:50:52.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 63981 6.58 MB/sec execute 401 sec latency 112.853 ms 2024-09-21T01:50:53.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 286964 29.37 MB/sec execute 400 sec latency 334.860 ms 2024-09-21T01:50:53.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:53 smithi090 ceph-mon[35142]: pgmap v893: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 118 op/s 2024-09-21T01:50:53.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:53 smithi045 ceph-mon[28361]: pgmap v893: 129 pgs: 129 active+clean; 444 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.7 MiB/s wr, 118 op/s 2024-09-21T01:50:53.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64102 6.59 MB/sec execute 402 sec latency 165.267 ms 2024-09-21T01:50:54.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 287209 29.33 MB/sec execute 401 sec latency 451.855 ms 2024-09-21T01:50:54.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64205 6.59 MB/sec execute 403 sec latency 182.608 ms 2024-09-21T01:50:55.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 287734 29.33 MB/sec execute 402 sec latency 496.600 ms 2024-09-21T01:50:55.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:55 smithi090 ceph-mon[35142]: pgmap v894: 129 pgs: 129 active+clean; 445 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.4 MiB/s wr, 96 op/s 2024-09-21T01:50:55.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:55 smithi045 ceph-mon[28361]: pgmap v894: 129 pgs: 129 active+clean; 445 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.4 MiB/s wr, 96 op/s 2024-09-21T01:50:55.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64306 6.59 MB/sec execute 404 sec latency 205.950 ms 2024-09-21T01:50:56.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 288116 29.30 MB/sec execute 403 sec latency 384.473 ms 2024-09-21T01:50:56.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64421 6.58 MB/sec execute 405 sec latency 211.168 ms 2024-09-21T01:50:57.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 288766 29.31 MB/sec execute 404 sec latency 101.001 ms 2024-09-21T01:50:57.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:57 smithi090 ceph-mon[35142]: pgmap v895: 129 pgs: 129 active+clean; 446 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 115 op/s 2024-09-21T01:50:57.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:57 smithi045 ceph-mon[28361]: pgmap v895: 129 pgs: 129 active+clean; 446 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.1 MiB/s wr, 115 op/s 2024-09-21T01:50:57.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64525 6.57 MB/sec execute 406 sec latency 134.568 ms 2024-09-21T01:50:58.712 INFO:tasks.workunit.client.0.smithi045.stdout: 10 290152 29.41 MB/sec execute 405 sec latency 116.472 ms 2024-09-21T01:50:58.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64650 6.56 MB/sec execute 407 sec latency 106.663 ms 2024-09-21T01:50:59.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 290478 29.40 MB/sec execute 406 sec latency 519.207 ms 2024-09-21T01:50:59.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:50:59 smithi090 ceph-mon[35142]: pgmap v896: 129 pgs: 129 active+clean; 448 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.5 MiB/s wr, 105 op/s 2024-09-21T01:50:59.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:50:59 smithi045 ceph-mon[28361]: pgmap v896: 129 pgs: 129 active+clean; 448 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.5 MiB/s wr, 105 op/s 2024-09-21T01:50:59.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64752 6.55 MB/sec execute 408 sec latency 175.424 ms 2024-09-21T01:51:00.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 290647 29.35 MB/sec execute 407 sec latency 715.084 ms 2024-09-21T01:51:00.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64860 6.55 MB/sec execute 409 sec latency 149.118 ms 2024-09-21T01:51:01.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:01 smithi090 ceph-mon[35142]: pgmap v897: 129 pgs: 129 active+clean; 447 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.1 MiB/s wr, 105 op/s 2024-09-21T01:51:01.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 291186 29.35 MB/sec execute 408 sec latency 547.260 ms 2024-09-21T01:51:01.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:01 smithi045 ceph-mon[28361]: pgmap v897: 129 pgs: 129 active+clean; 447 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.1 MiB/s wr, 105 op/s 2024-09-21T01:51:01.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 64963 6.54 MB/sec execute 410 sec latency 237.204 ms 2024-09-21T01:51:02.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 291292 29.29 MB/sec execute 409 sec latency 509.749 ms 2024-09-21T01:51:02.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 65088 6.54 MB/sec execute 411 sec latency 156.904 ms 2024-09-21T01:51:03.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 291597 29.26 MB/sec execute 410 sec latency 515.735 ms 2024-09-21T01:51:03.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:03 smithi090 ceph-mon[35142]: pgmap v898: 129 pgs: 129 active+clean; 441 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.1 MiB/s wr, 119 op/s 2024-09-21T01:51:03.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:03 smithi045 ceph-mon[28361]: pgmap v898: 129 pgs: 129 active+clean; 441 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.1 MiB/s wr, 119 op/s 2024-09-21T01:51:03.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 65215 6.54 MB/sec execute 412 sec latency 194.778 ms 2024-09-21T01:51:04.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 291829 29.21 MB/sec execute 411 sec latency 84.253 ms 2024-09-21T01:51:04.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 65368 6.54 MB/sec execute 413 sec latency 91.746 ms 2024-09-21T01:51:05.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 292971 29.26 MB/sec execute 412 sec latency 373.328 ms 2024-09-21T01:51:05.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:05 smithi090 ceph-mon[35142]: pgmap v899: 129 pgs: 129 active+clean; 438 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 MiB/s wr, 114 op/s 2024-09-21T01:51:05.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:05 smithi045 ceph-mon[28361]: pgmap v899: 129 pgs: 129 active+clean; 438 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 MiB/s wr, 114 op/s 2024-09-21T01:51:05.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 65537 6.55 MB/sec execute 414 sec latency 101.789 ms 2024-09-21T01:51:06.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 294191 29.39 MB/sec execute 413 sec latency 102.893 ms 2024-09-21T01:51:06.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 65700 6.55 MB/sec execute 415 sec latency 110.409 ms 2024-09-21T01:51:07.669 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:07 smithi045 ceph-mon[28361]: pgmap v900: 129 pgs: 129 active+clean; 434 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-09-21T01:51:07.670 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:07.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 295153 29.44 MB/sec execute 414 sec latency 123.171 ms 2024-09-21T01:51:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:07 smithi090 ceph-mon[35142]: pgmap v900: 129 pgs: 129 active+clean; 434 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.2 MiB/s wr, 172 op/s 2024-09-21T01:51:07.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:07.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 65857 6.54 MB/sec execute 416 sec latency 78.077 ms 2024-09-21T01:51:08.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 295425 29.39 MB/sec execute 415 sec latency 95.813 ms 2024-09-21T01:51:08.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66039 6.54 MB/sec execute 417 sec latency 96.146 ms 2024-09-21T01:51:09.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 296806 29.48 MB/sec execute 416 sec latency 103.161 ms 2024-09-21T01:51:09.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:09 smithi090 ceph-mon[35142]: pgmap v901: 129 pgs: 129 active+clean; 432 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.6 MiB/s wr, 167 op/s 2024-09-21T01:51:09.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:09 smithi045 ceph-mon[28361]: pgmap v901: 129 pgs: 129 active+clean; 432 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.6 MiB/s wr, 167 op/s 2024-09-21T01:51:09.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66202 6.55 MB/sec execute 418 sec latency 105.216 ms 2024-09-21T01:51:10.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 298377 29.65 MB/sec execute 417 sec latency 109.197 ms 2024-09-21T01:51:10.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66360 6.55 MB/sec execute 419 sec latency 104.256 ms 2024-09-21T01:51:11.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:11 smithi090 ceph-mon[35142]: pgmap v902: 129 pgs: 129 active+clean; 435 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-09-21T01:51:11.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 298757 29.62 MB/sec execute 418 sec latency 101.254 ms 2024-09-21T01:51:11.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:11 smithi045 ceph-mon[28361]: pgmap v902: 129 pgs: 129 active+clean; 435 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.5 MiB/s wr, 173 op/s 2024-09-21T01:51:11.952 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66527 6.56 MB/sec execute 420 sec latency 88.185 ms 2024-09-21T01:51:12.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 299407 29.63 MB/sec execute 419 sec latency 142.255 ms 2024-09-21T01:51:12.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66676 6.57 MB/sec execute 421 sec latency 98.308 ms 2024-09-21T01:51:13.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 300800 29.73 MB/sec execute 420 sec latency 112.984 ms 2024-09-21T01:51:13.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:13 smithi090 ceph-mon[35142]: pgmap v903: 129 pgs: 129 active+clean; 439 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-09-21T01:51:13.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:13 smithi045 ceph-mon[28361]: pgmap v903: 129 pgs: 129 active+clean; 439 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 189 op/s 2024-09-21T01:51:13.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66824 6.58 MB/sec execute 422 sec latency 103.264 ms 2024-09-21T01:51:14.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 302210 29.86 MB/sec execute 421 sec latency 104.952 ms 2024-09-21T01:51:14.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 66969 6.60 MB/sec execute 423 sec latency 120.681 ms 2024-09-21T01:51:15.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 302309 29.79 MB/sec execute 422 sec latency 275.496 ms 2024-09-21T01:51:15.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:15 smithi090 ceph-mon[35142]: pgmap v904: 129 pgs: 129 active+clean; 441 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.4 MiB/s wr, 153 op/s 2024-09-21T01:51:15.748 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:15 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:51:15.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:15 smithi045 ceph-mon[28361]: pgmap v904: 129 pgs: 129 active+clean; 441 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.4 MiB/s wr, 153 op/s 2024-09-21T01:51:15.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:15 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:51:15.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67111 6.61 MB/sec execute 424 sec latency 158.838 ms 2024-09-21T01:51:16.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:51:16.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:51:16.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:51:16.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 302992 29.80 MB/sec execute 423 sec latency 101.539 ms 2024-09-21T01:51:16.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:51:16.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:51:16.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:51:16.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67252 6.62 MB/sec execute 425 sec latency 112.639 ms 2024-09-21T01:51:17.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 304359 29.90 MB/sec execute 424 sec latency 109.863 ms 2024-09-21T01:51:17.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67368 6.62 MB/sec execute 426 sec latency 118.414 ms 2024-09-21T01:51:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:17 smithi045 ceph-mon[28361]: pgmap v905: 129 pgs: 129 active+clean; 450 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 192 op/s 2024-09-21T01:51:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:17 smithi090 ceph-mon[35142]: pgmap v905: 129 pgs: 129 active+clean; 450 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 3.6 MiB/s wr, 192 op/s 2024-09-21T01:51:18.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 305755 30.03 MB/sec execute 425 sec latency 115.686 ms 2024-09-21T01:51:18.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67464 6.62 MB/sec execute 427 sec latency 120.743 ms 2024-09-21T01:51:19.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 305849 29.96 MB/sec execute 426 sec latency 363.186 ms 2024-09-21T01:51:19.953 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67542 6.61 MB/sec execute 428 sec latency 130.990 ms 2024-09-21T01:51:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:19 smithi045 ceph-mon[28361]: pgmap v906: 129 pgs: 129 active+clean; 456 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-09-21T01:51:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:19 smithi090 ceph-mon[35142]: pgmap v906: 129 pgs: 129 active+clean; 456 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.6 MiB/s wr, 140 op/s 2024-09-21T01:51:20.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 306139 29.92 MB/sec execute 427 sec latency 262.864 ms 2024-09-21T01:51:20.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67653 6.62 MB/sec execute 429 sec latency 172.181 ms 2024-09-21T01:51:21.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 307435 29.99 MB/sec execute 428 sec latency 211.263 ms 2024-09-21T01:51:21.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67795 6.63 MB/sec execute 430 sec latency 115.878 ms 2024-09-21T01:51:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:21 smithi045 ceph-mon[28361]: pgmap v907: 129 pgs: 129 active+clean; 457 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.8 MiB/s wr, 128 op/s 2024-09-21T01:51:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:21 smithi090 ceph-mon[35142]: pgmap v907: 129 pgs: 129 active+clean; 457 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.8 MiB/s wr, 128 op/s 2024-09-21T01:51:22.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 309023 30.16 MB/sec execute 429 sec latency 100.698 ms 2024-09-21T01:51:22.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 67916 6.62 MB/sec execute 431 sec latency 107.909 ms 2024-09-21T01:51:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:23.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:23.713 INFO:tasks.workunit.client.0.smithi045.stdout: 10 309426 30.13 MB/sec execute 430 sec latency 86.826 ms 2024-09-21T01:51:23.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68033 6.61 MB/sec execute 432 sec latency 128.039 ms 2024-09-21T01:51:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:23 smithi045 ceph-mon[28361]: pgmap v908: 129 pgs: 129 active+clean; 460 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.1 MiB/s wr, 126 op/s 2024-09-21T01:51:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:23 smithi090 ceph-mon[35142]: pgmap v908: 129 pgs: 129 active+clean; 460 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.1 MiB/s wr, 126 op/s 2024-09-21T01:51:24.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 310209 30.14 MB/sec execute 431 sec latency 181.419 ms 2024-09-21T01:51:24.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68139 6.61 MB/sec execute 433 sec latency 199.396 ms 2024-09-21T01:51:25.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 310709 30.12 MB/sec execute 432 sec latency 432.967 ms 2024-09-21T01:51:25.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68186 6.60 MB/sec execute 434 sec latency 293.920 ms 2024-09-21T01:51:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:25 smithi045 ceph-mon[28361]: pgmap v909: 129 pgs: 129 active+clean; 462 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-09-21T01:51:26.239 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:25 smithi090 ceph-mon[35142]: pgmap v909: 129 pgs: 129 active+clean; 462 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-09-21T01:51:26.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 310904 30.07 MB/sec execute 433 sec latency 475.830 ms 2024-09-21T01:51:26.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68279 6.58 MB/sec execute 435 sec latency 206.114 ms 2024-09-21T01:51:27.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 311268 30.06 MB/sec execute 434 sec latency 368.127 ms 2024-09-21T01:51:27.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68354 6.58 MB/sec execute 436 sec latency 223.508 ms 2024-09-21T01:51:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:27 smithi045 ceph-mon[28361]: pgmap v910: 129 pgs: 129 active+clean; 434 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.4 MiB/s wr, 137 op/s 2024-09-21T01:51:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:27 smithi090 ceph-mon[35142]: pgmap v910: 129 pgs: 129 active+clean; 434 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.4 MiB/s wr, 137 op/s 2024-09-21T01:51:28.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 311483 30.03 MB/sec execute 435 sec latency 354.470 ms 2024-09-21T01:51:28.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68456 6.57 MB/sec execute 437 sec latency 195.876 ms 2024-09-21T01:51:29.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 311821 30.02 MB/sec execute 436 sec latency 368.017 ms 2024-09-21T01:51:29.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68552 6.56 MB/sec execute 438 sec latency 170.487 ms 2024-09-21T01:51:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:29 smithi045 ceph-mon[28361]: pgmap v911: 129 pgs: 129 active+clean; 433 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 119 op/s 2024-09-21T01:51:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:29 smithi090 ceph-mon[35142]: pgmap v911: 129 pgs: 129 active+clean; 433 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.4 KiB/s rd, 3.5 MiB/s wr, 119 op/s 2024-09-21T01:51:30.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 311996 29.97 MB/sec execute 437 sec latency 430.711 ms 2024-09-21T01:51:30.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68654 6.56 MB/sec execute 439 sec latency 212.793 ms 2024-09-21T01:51:31.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312558 29.97 MB/sec execute 438 sec latency 537.209 ms 2024-09-21T01:51:31.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68770 6.56 MB/sec execute 440 sec latency 165.026 ms 2024-09-21T01:51:32.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:31 smithi045 ceph-mon[28361]: pgmap v912: 129 pgs: 129 active+clean; 431 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.0 MiB/s wr, 123 op/s 2024-09-21T01:51:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:31 smithi090 ceph-mon[35142]: pgmap v912: 129 pgs: 129 active+clean; 431 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.0 MiB/s wr, 123 op/s 2024-09-21T01:51:32.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312577 29.91 MB/sec execute 439 sec latency 593.583 ms 2024-09-21T01:51:32.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68867 6.55 MB/sec execute 441 sec latency 172.408 ms 2024-09-21T01:51:33.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312867 29.87 MB/sec execute 440 sec latency 465.675 ms 2024-09-21T01:51:33.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 68978 6.54 MB/sec execute 442 sec latency 174.851 ms 2024-09-21T01:51:34.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:33 smithi045 ceph-mon[28361]: pgmap v913: 129 pgs: 129 active+clean; 425 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.0 MiB/s wr, 160 op/s 2024-09-21T01:51:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:33 smithi090 ceph-mon[35142]: pgmap v913: 129 pgs: 129 active+clean; 425 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.0 MiB/s wr, 160 op/s 2024-09-21T01:51:34.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312878 29.81 MB/sec execute 441 sec latency 418.995 ms 2024-09-21T01:51:34.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69112 6.55 MB/sec execute 443 sec latency 167.086 ms 2024-09-21T01:51:35.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312901 29.74 MB/sec execute 442 sec latency 417.412 ms 2024-09-21T01:51:35.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69246 6.55 MB/sec execute 444 sec latency 194.027 ms 2024-09-21T01:51:36.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:35 smithi045 ceph-mon[28361]: pgmap v914: 129 pgs: 129 active+clean; 422 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.2 MiB/s wr, 127 op/s 2024-09-21T01:51:36.239 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:35 smithi090 ceph-mon[35142]: pgmap v914: 129 pgs: 129 active+clean; 422 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.2 MiB/s wr, 127 op/s 2024-09-21T01:51:36.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312941 29.67 MB/sec execute 443 sec latency 417.462 ms 2024-09-21T01:51:36.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69385 6.55 MB/sec execute 445 sec latency 169.986 ms 2024-09-21T01:51:37.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 312952 29.61 MB/sec execute 444 sec latency 394.711 ms 2024-09-21T01:51:37.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69542 6.55 MB/sec execute 446 sec latency 146.889 ms 2024-09-21T01:51:38.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:37 smithi045 ceph-mon[28361]: pgmap v915: 129 pgs: 129 active+clean; 420 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-09-21T01:51:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:37 smithi090 ceph-mon[35142]: pgmap v915: 129 pgs: 129 active+clean; 420 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 2.5 MiB/s wr, 152 op/s 2024-09-21T01:51:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:38.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 313013 29.55 MB/sec execute 445 sec latency 535.223 ms 2024-09-21T01:51:38.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69693 6.55 MB/sec execute 447 sec latency 138.841 ms 2024-09-21T01:51:39.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 313052 29.48 MB/sec execute 446 sec latency 406.744 ms 2024-09-21T01:51:39.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69794 6.54 MB/sec execute 448 sec latency 177.594 ms 2024-09-21T01:51:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:39 smithi045 ceph-mon[28361]: pgmap v916: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-09-21T01:51:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:39 smithi090 ceph-mon[35142]: pgmap v916: 129 pgs: 129 active+clean; 421 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.0 MiB/s wr, 154 op/s 2024-09-21T01:51:40.714 INFO:tasks.workunit.client.0.smithi045.stdout: 10 313181 29.43 MB/sec execute 447 sec latency 473.907 ms 2024-09-21T01:51:40.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 69923 6.54 MB/sec execute 449 sec latency 186.261 ms 2024-09-21T01:51:41.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 314315 29.48 MB/sec execute 448 sec latency 417.681 ms 2024-09-21T01:51:41.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70057 6.54 MB/sec execute 450 sec latency 133.423 ms 2024-09-21T01:51:42.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:41 smithi045 ceph-mon[28361]: pgmap v917: 129 pgs: 129 active+clean; 422 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-09-21T01:51:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:41 smithi090 ceph-mon[35142]: pgmap v917: 129 pgs: 129 active+clean; 422 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 1.9 MiB/s wr, 136 op/s 2024-09-21T01:51:42.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 315830 29.63 MB/sec execute 449 sec latency 163.926 ms 2024-09-21T01:51:42.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70215 6.56 MB/sec execute 451 sec latency 110.670 ms 2024-09-21T01:51:43.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 316437 29.64 MB/sec execute 450 sec latency 121.498 ms 2024-09-21T01:51:43.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70365 6.56 MB/sec execute 452 sec latency 115.273 ms 2024-09-21T01:51:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:43 smithi045 ceph-mon[28361]: pgmap v918: 129 pgs: 129 active+clean; 428 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-09-21T01:51:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:43 smithi090 ceph-mon[35142]: pgmap v918: 129 pgs: 129 active+clean; 428 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.4 MiB/s wr, 163 op/s 2024-09-21T01:51:44.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 316685 29.59 MB/sec execute 451 sec latency 99.190 ms 2024-09-21T01:51:44.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70528 6.59 MB/sec execute 453 sec latency 119.368 ms 2024-09-21T01:51:45.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 318019 29.66 MB/sec execute 452 sec latency 139.980 ms 2024-09-21T01:51:45.954 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70705 6.60 MB/sec execute 454 sec latency 105.692 ms 2024-09-21T01:51:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:45 smithi045 ceph-mon[28361]: pgmap v919: 129 pgs: 129 active+clean; 429 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-09-21T01:51:46.239 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:45 smithi090 ceph-mon[35142]: pgmap v919: 129 pgs: 129 active+clean; 429 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.1 MiB/s wr, 122 op/s 2024-09-21T01:51:46.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 319024 29.75 MB/sec execute 453 sec latency 199.804 ms 2024-09-21T01:51:46.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70798 6.60 MB/sec execute 455 sec latency 232.982 ms 2024-09-21T01:51:47.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 319646 29.76 MB/sec execute 454 sec latency 537.026 ms 2024-09-21T01:51:47.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70914 6.61 MB/sec execute 456 sec latency 179.964 ms 2024-09-21T01:51:48.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:47 smithi045 ceph-mon[28361]: pgmap v920: 129 pgs: 129 active+clean; 438 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.2 MiB/s wr, 136 op/s 2024-09-21T01:51:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:47 smithi090 ceph-mon[35142]: pgmap v920: 129 pgs: 129 active+clean; 438 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.2 MiB/s wr, 136 op/s 2024-09-21T01:51:48.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 319664 29.69 MB/sec execute 455 sec latency 519.827 ms 2024-09-21T01:51:48.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 70975 6.60 MB/sec execute 457 sec latency 189.170 ms 2024-09-21T01:51:49.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 319954 29.66 MB/sec execute 456 sec latency 483.926 ms 2024-09-21T01:51:49.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71031 6.59 MB/sec execute 458 sec latency 205.268 ms 2024-09-21T01:51:50.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:49 smithi045 ceph-mon[28361]: pgmap v921: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.3 MiB/s wr, 124 op/s 2024-09-21T01:51:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:49 smithi090 ceph-mon[35142]: pgmap v921: 129 pgs: 129 active+clean; 443 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.3 MiB/s wr, 124 op/s 2024-09-21T01:51:50.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 319969 29.60 MB/sec execute 457 sec latency 409.010 ms 2024-09-21T01:51:50.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71086 6.58 MB/sec execute 459 sec latency 231.915 ms 2024-09-21T01:51:51.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 319985 29.54 MB/sec execute 458 sec latency 417.032 ms 2024-09-21T01:51:51.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71148 6.58 MB/sec execute 460 sec latency 194.949 ms 2024-09-21T01:51:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:51 smithi045 ceph-mon[28361]: pgmap v922: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 106 op/s 2024-09-21T01:51:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:51 smithi090 ceph-mon[35142]: pgmap v922: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 106 op/s 2024-09-21T01:51:52.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 320029 29.47 MB/sec execute 459 sec latency 505.561 ms 2024-09-21T01:51:52.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71233 6.58 MB/sec execute 461 sec latency 253.006 ms 2024-09-21T01:51:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:51:53.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 320036 29.41 MB/sec execute 460 sec latency 377.874 ms 2024-09-21T01:51:53.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71338 6.58 MB/sec execute 462 sec latency 163.549 ms 2024-09-21T01:51:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:53 smithi045 ceph-mon[28361]: pgmap v923: 129 pgs: 129 active+clean; 447 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.9 MiB/s wr, 122 op/s 2024-09-21T01:51:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:53 smithi090 ceph-mon[35142]: pgmap v923: 129 pgs: 129 active+clean; 447 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 3.9 MiB/s wr, 122 op/s 2024-09-21T01:51:54.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 320097 29.35 MB/sec execute 461 sec latency 409.005 ms 2024-09-21T01:51:54.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71434 6.57 MB/sec execute 463 sec latency 160.450 ms 2024-09-21T01:51:55.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 320696 29.36 MB/sec execute 462 sec latency 350.796 ms 2024-09-21T01:51:55.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71542 6.57 MB/sec execute 464 sec latency 184.160 ms 2024-09-21T01:51:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:55 smithi045 ceph-mon[28361]: pgmap v924: 129 pgs: 129 active+clean; 447 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 86 op/s 2024-09-21T01:51:56.240 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:55 smithi090 ceph-mon[35142]: pgmap v924: 129 pgs: 129 active+clean; 447 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 86 op/s 2024-09-21T01:51:56.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 322096 29.45 MB/sec execute 463 sec latency 111.698 ms 2024-09-21T01:51:56.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71665 6.56 MB/sec execute 465 sec latency 112.430 ms 2024-09-21T01:51:57.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 323483 29.56 MB/sec execute 464 sec latency 107.895 ms 2024-09-21T01:51:57.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71773 6.55 MB/sec execute 466 sec latency 132.733 ms 2024-09-21T01:51:58.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:57 smithi045 ceph-mon[28361]: pgmap v925: 129 pgs: 129 active+clean; 447 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.7 MiB/s wr, 94 op/s 2024-09-21T01:51:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:57 smithi090 ceph-mon[35142]: pgmap v925: 129 pgs: 129 active+clean; 447 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.7 MiB/s wr, 94 op/s 2024-09-21T01:51:58.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 323643 29.51 MB/sec execute 465 sec latency 133.874 ms 2024-09-21T01:51:58.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 71902 6.55 MB/sec execute 467 sec latency 158.592 ms 2024-09-21T01:51:59.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 324463 29.52 MB/sec execute 466 sec latency 129.247 ms 2024-09-21T01:51:59.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72033 6.54 MB/sec execute 468 sec latency 133.571 ms 2024-09-21T01:52:00.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:51:59 smithi045 ceph-mon[28361]: pgmap v926: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.1 MiB/s wr, 115 op/s 2024-09-21T01:52:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:51:59 smithi090 ceph-mon[35142]: pgmap v926: 129 pgs: 129 active+clean; 450 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.1 MiB/s wr, 115 op/s 2024-09-21T01:52:00.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 325983 29.66 MB/sec execute 467 sec latency 105.778 ms 2024-09-21T01:52:00.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72202 6.55 MB/sec execute 469 sec latency 101.496 ms 2024-09-21T01:52:01.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 327062 29.72 MB/sec execute 468 sec latency 132.209 ms 2024-09-21T01:52:01.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72351 6.55 MB/sec execute 470 sec latency 104.906 ms 2024-09-21T01:52:02.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:01 smithi045 ceph-mon[28361]: pgmap v927: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.9 MiB/s wr, 96 op/s 2024-09-21T01:52:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:01 smithi090 ceph-mon[35142]: pgmap v927: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.9 MiB/s wr, 96 op/s 2024-09-21T01:52:02.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 327261 29.67 MB/sec execute 469 sec latency 100.588 ms 2024-09-21T01:52:02.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72506 6.55 MB/sec execute 471 sec latency 103.612 ms 2024-09-21T01:52:03.715 INFO:tasks.workunit.client.0.smithi045.stdout: 10 328656 29.74 MB/sec execute 470 sec latency 205.434 ms 2024-09-21T01:52:03.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72675 6.56 MB/sec execute 472 sec latency 116.619 ms 2024-09-21T01:52:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:03 smithi045 ceph-mon[28361]: pgmap v928: 129 pgs: 129 active+clean; 436 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.7 MiB/s wr, 131 op/s 2024-09-21T01:52:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:03 smithi090 ceph-mon[35142]: pgmap v928: 129 pgs: 129 active+clean; 436 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.7 MiB/s wr, 131 op/s 2024-09-21T01:52:04.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 329585 29.83 MB/sec execute 471 sec latency 360.050 ms 2024-09-21T01:52:04.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72827 6.56 MB/sec execute 473 sec latency 101.348 ms 2024-09-21T01:52:05.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330602 29.87 MB/sec execute 472 sec latency 352.350 ms 2024-09-21T01:52:05.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 72948 6.55 MB/sec execute 474 sec latency 272.154 ms 2024-09-21T01:52:06.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:05 smithi045 ceph-mon[28361]: pgmap v929: 129 pgs: 129 active+clean; 436 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-09-21T01:52:06.240 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:05 smithi090 ceph-mon[35142]: pgmap v929: 129 pgs: 129 active+clean; 436 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.2 MiB/s wr, 120 op/s 2024-09-21T01:52:06.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330608 29.81 MB/sec execute 473 sec latency 452.249 ms 2024-09-21T01:52:06.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73073 6.55 MB/sec execute 475 sec latency 187.806 ms 2024-09-21T01:52:07.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330625 29.75 MB/sec execute 474 sec latency 408.758 ms 2024-09-21T01:52:07.955 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73172 6.54 MB/sec execute 476 sec latency 272.130 ms 2024-09-21T01:52:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:07 smithi045 ceph-mon[28361]: pgmap v930: 129 pgs: 129 active+clean; 415 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-09-21T01:52:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:07 smithi090 ceph-mon[35142]: pgmap v930: 129 pgs: 129 active+clean; 415 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.0 MiB/s wr, 178 op/s 2024-09-21T01:52:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:08.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330671 29.69 MB/sec execute 475 sec latency 400.941 ms 2024-09-21T01:52:08.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73302 6.54 MB/sec execute 477 sec latency 282.010 ms 2024-09-21T01:52:09.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330684 29.63 MB/sec execute 476 sec latency 433.954 ms 2024-09-21T01:52:09.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73386 6.53 MB/sec execute 478 sec latency 194.840 ms 2024-09-21T01:52:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:09 smithi045 ceph-mon[28361]: pgmap v931: 129 pgs: 129 active+clean; 412 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.7 MiB/s wr, 182 op/s 2024-09-21T01:52:10.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:09 smithi090 ceph-mon[35142]: pgmap v931: 129 pgs: 129 active+clean; 412 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.7 MiB/s wr, 182 op/s 2024-09-21T01:52:10.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330749 29.57 MB/sec execute 477 sec latency 442.198 ms 2024-09-21T01:52:10.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73510 6.53 MB/sec execute 479 sec latency 197.028 ms 2024-09-21T01:52:11.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 330783 29.51 MB/sec execute 478 sec latency 447.800 ms 2024-09-21T01:52:11.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73661 6.54 MB/sec execute 480 sec latency 113.274 ms 2024-09-21T01:52:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:11 smithi045 ceph-mon[28361]: pgmap v932: 129 pgs: 129 active+clean; 413 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-09-21T01:52:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:11 smithi090 ceph-mon[35142]: pgmap v932: 129 pgs: 129 active+clean; 413 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.3 MiB/s wr, 151 op/s 2024-09-21T01:52:12.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 331107 29.49 MB/sec execute 479 sec latency 548.717 ms 2024-09-21T01:52:12.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73782 6.54 MB/sec execute 481 sec latency 187.887 ms 2024-09-21T01:52:13.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 331304 29.45 MB/sec execute 480 sec latency 524.624 ms 2024-09-21T01:52:13.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73885 6.55 MB/sec execute 482 sec latency 190.769 ms 2024-09-21T01:52:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:13 smithi045 ceph-mon[28361]: pgmap v933: 129 pgs: 129 active+clean; 420 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-09-21T01:52:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:13 smithi090 ceph-mon[35142]: pgmap v933: 129 pgs: 129 active+clean; 420 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-09-21T01:52:14.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 331495 29.39 MB/sec execute 481 sec latency 576.170 ms 2024-09-21T01:52:14.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 73971 6.55 MB/sec execute 483 sec latency 182.620 ms 2024-09-21T01:52:15.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 331884 29.37 MB/sec execute 482 sec latency 517.808 ms 2024-09-21T01:52:15.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74081 6.56 MB/sec execute 484 sec latency 217.112 ms 2024-09-21T01:52:16.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:15 smithi045 ceph-mon[28361]: pgmap v934: 129 pgs: 129 active+clean; 422 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-09-21T01:52:16.241 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:15 smithi090 ceph-mon[35142]: pgmap v934: 129 pgs: 129 active+clean; 422 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 2.5 MiB/s wr, 140 op/s 2024-09-21T01:52:16.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 332693 29.41 MB/sec execute 483 sec latency 500.573 ms 2024-09-21T01:52:16.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74213 6.57 MB/sec execute 485 sec latency 227.186 ms 2024-09-21T01:52:17.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:16 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:52:17.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:16 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:52:17.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 334135 29.53 MB/sec execute 484 sec latency 104.150 ms 2024-09-21T01:52:17.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74360 6.57 MB/sec execute 486 sec latency 98.470 ms 2024-09-21T01:52:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:17 smithi045 ceph-mon[28361]: pgmap v935: 129 pgs: 129 active+clean; 426 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-09-21T01:52:18.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:17 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:52:18.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:17 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:52:18.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:17 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:52:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:17 smithi090 ceph-mon[35142]: pgmap v935: 129 pgs: 129 active+clean; 426 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-09-21T01:52:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:17 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:52:18.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:17 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:52:18.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:17 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:52:18.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 334217 29.47 MB/sec execute 485 sec latency 306.238 ms 2024-09-21T01:52:18.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74500 6.58 MB/sec execute 487 sec latency 109.594 ms 2024-09-21T01:52:19.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 335051 29.48 MB/sec execute 486 sec latency 94.815 ms 2024-09-21T01:52:19.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74598 6.58 MB/sec execute 488 sec latency 100.546 ms 2024-09-21T01:52:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:19 smithi045 ceph-mon[28361]: pgmap v936: 129 pgs: 129 active+clean; 431 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.9 MiB/s wr, 107 op/s 2024-09-21T01:52:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:19 smithi090 ceph-mon[35142]: pgmap v936: 129 pgs: 129 active+clean; 431 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.9 MiB/s wr, 107 op/s 2024-09-21T01:52:20.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 336629 29.62 MB/sec execute 487 sec latency 105.817 ms 2024-09-21T01:52:20.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74676 6.57 MB/sec execute 489 sec latency 110.631 ms 2024-09-21T01:52:21.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 337700 29.68 MB/sec execute 488 sec latency 128.816 ms 2024-09-21T01:52:21.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74781 6.58 MB/sec execute 490 sec latency 209.950 ms 2024-09-21T01:52:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:21 smithi045 ceph-mon[28361]: pgmap v937: 129 pgs: 129 active+clean; 436 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.4 MiB/s wr, 104 op/s 2024-09-21T01:52:22.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:21 smithi090 ceph-mon[35142]: pgmap v937: 129 pgs: 129 active+clean; 436 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 3.4 MiB/s wr, 104 op/s 2024-09-21T01:52:22.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 337874 29.63 MB/sec execute 489 sec latency 108.395 ms 2024-09-21T01:52:22.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 74919 6.58 MB/sec execute 491 sec latency 113.567 ms 2024-09-21T01:52:23.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:23.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:23.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 339157 29.68 MB/sec execute 490 sec latency 217.772 ms 2024-09-21T01:52:23.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75041 6.58 MB/sec execute 492 sec latency 117.202 ms 2024-09-21T01:52:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:23 smithi045 ceph-mon[28361]: pgmap v938: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 112 op/s 2024-09-21T01:52:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:23 smithi090 ceph-mon[35142]: pgmap v938: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 112 op/s 2024-09-21T01:52:24.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 340204 29.77 MB/sec execute 491 sec latency 121.978 ms 2024-09-21T01:52:24.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75158 6.58 MB/sec execute 493 sec latency 138.898 ms 2024-09-21T01:52:25.716 INFO:tasks.workunit.client.0.smithi045.stdout: 10 341244 29.83 MB/sec execute 492 sec latency 111.321 ms 2024-09-21T01:52:25.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75272 6.57 MB/sec execute 494 sec latency 119.223 ms 2024-09-21T01:52:26.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:25 smithi045 ceph-mon[28361]: pgmap v939: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.1 MiB/s wr, 75 op/s 2024-09-21T01:52:26.241 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:25 smithi090 ceph-mon[35142]: pgmap v939: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.1 MiB/s wr, 75 op/s 2024-09-21T01:52:26.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 341407 29.77 MB/sec execute 493 sec latency 222.112 ms 2024-09-21T01:52:26.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75374 6.56 MB/sec execute 495 sec latency 158.899 ms 2024-09-21T01:52:27.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 341459 29.71 MB/sec execute 494 sec latency 367.386 ms 2024-09-21T01:52:27.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75448 6.56 MB/sec execute 496 sec latency 194.616 ms 2024-09-21T01:52:28.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:27 smithi045 ceph-mon[28361]: pgmap v940: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.1 MiB/s wr, 127 op/s 2024-09-21T01:52:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:27 smithi090 ceph-mon[35142]: pgmap v940: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.1 MiB/s wr, 127 op/s 2024-09-21T01:52:28.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 341938 29.72 MB/sec execute 495 sec latency 394.619 ms 2024-09-21T01:52:28.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75552 6.55 MB/sec execute 497 sec latency 213.175 ms 2024-09-21T01:52:29.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 342139 29.66 MB/sec execute 496 sec latency 483.727 ms 2024-09-21T01:52:29.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75638 6.54 MB/sec execute 498 sec latency 188.001 ms 2024-09-21T01:52:30.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:29 smithi045 ceph-mon[28361]: pgmap v941: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.6 MiB/s wr, 124 op/s 2024-09-21T01:52:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:29 smithi090 ceph-mon[35142]: pgmap v941: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.6 MiB/s wr, 124 op/s 2024-09-21T01:52:30.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 342640 29.65 MB/sec execute 497 sec latency 461.245 ms 2024-09-21T01:52:30.956 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75761 6.54 MB/sec execute 499 sec latency 192.764 ms 2024-09-21T01:52:31.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 342746 29.59 MB/sec execute 498 sec latency 567.304 ms 2024-09-21T01:52:31.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 75876 6.54 MB/sec execute 500 sec latency 176.480 ms 2024-09-21T01:52:32.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:31 smithi045 ceph-mon[28361]: pgmap v942: 129 pgs: 129 active+clean; 434 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 MiB/s wr, 110 op/s 2024-09-21T01:52:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:31 smithi090 ceph-mon[35142]: pgmap v942: 129 pgs: 129 active+clean; 434 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.2 MiB/s wr, 110 op/s 2024-09-21T01:52:32.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 342939 29.57 MB/sec execute 499 sec latency 560.514 ms 2024-09-21T01:52:32.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76004 6.53 MB/sec execute 501 sec latency 155.886 ms 2024-09-21T01:52:33.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 343316 29.56 MB/sec execute 500 sec latency 556.793 ms 2024-09-21T01:52:33.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76137 6.53 MB/sec execute 502 sec latency 174.826 ms 2024-09-21T01:52:34.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:33 smithi045 ceph-mon[28361]: pgmap v943: 129 pgs: 129 active+clean; 429 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-09-21T01:52:34.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:33 smithi090 ceph-mon[35142]: pgmap v943: 129 pgs: 129 active+clean; 429 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.0 MiB/s wr, 151 op/s 2024-09-21T01:52:34.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 343537 29.52 MB/sec execute 501 sec latency 593.311 ms 2024-09-21T01:52:34.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76232 6.53 MB/sec execute 503 sec latency 188.155 ms 2024-09-21T01:52:35.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 343731 29.50 MB/sec execute 502 sec latency 411.963 ms 2024-09-21T01:52:35.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76325 6.52 MB/sec execute 504 sec latency 200.558 ms 2024-09-21T01:52:36.241 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:35 smithi090 ceph-mon[35142]: pgmap v944: 129 pgs: 129 active+clean; 427 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.4 MiB/s wr, 139 op/s 2024-09-21T01:52:36.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:35 smithi045 ceph-mon[28361]: pgmap v944: 129 pgs: 129 active+clean; 427 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.4 MiB/s wr, 139 op/s 2024-09-21T01:52:36.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 344415 29.51 MB/sec execute 503 sec latency 458.032 ms 2024-09-21T01:52:36.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76446 6.52 MB/sec execute 505 sec latency 165.472 ms 2024-09-21T01:52:37.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 344473 29.46 MB/sec execute 504 sec latency 488.598 ms 2024-09-21T01:52:37.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76631 6.52 MB/sec execute 506 sec latency 114.786 ms 2024-09-21T01:52:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:37 smithi090 ceph-mon[35142]: pgmap v945: 129 pgs: 129 active+clean; 424 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-09-21T01:52:38.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:38.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:37 smithi045 ceph-mon[28361]: pgmap v945: 129 pgs: 129 active+clean; 424 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.8 MiB/s wr, 163 op/s 2024-09-21T01:52:38.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:38.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 344853 29.44 MB/sec execute 505 sec latency 385.877 ms 2024-09-21T01:52:38.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76804 6.53 MB/sec execute 507 sec latency 99.324 ms 2024-09-21T01:52:39.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 345492 29.45 MB/sec execute 506 sec latency 105.751 ms 2024-09-21T01:52:39.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 76936 6.52 MB/sec execute 508 sec latency 95.630 ms 2024-09-21T01:52:40.207 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:39 smithi045 ceph-mon[28361]: pgmap v946: 129 pgs: 129 active+clean; 427 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.1 MiB/s wr, 146 op/s 2024-09-21T01:52:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:39 smithi090 ceph-mon[35142]: pgmap v946: 129 pgs: 129 active+clean; 427 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 2.1 MiB/s wr, 146 op/s 2024-09-21T01:52:40.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 346871 29.53 MB/sec execute 507 sec latency 113.539 ms 2024-09-21T01:52:40.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 77101 6.52 MB/sec execute 509 sec latency 87.023 ms 2024-09-21T01:52:41.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 348164 29.63 MB/sec execute 508 sec latency 106.359 ms 2024-09-21T01:52:41.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 77269 6.54 MB/sec execute 510 sec latency 91.206 ms 2024-09-21T01:52:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:41 smithi090 ceph-mon[35142]: pgmap v947: 129 pgs: 129 active+clean; 428 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-21T01:52:42.421 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:41 smithi045 ceph-mon[28361]: pgmap v947: 129 pgs: 129 active+clean; 428 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 143 op/s 2024-09-21T01:52:42.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 348413 29.58 MB/sec execute 509 sec latency 94.296 ms 2024-09-21T01:52:42.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 77434 6.55 MB/sec execute 511 sec latency 105.409 ms 2024-09-21T01:52:43.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 349282 29.60 MB/sec execute 510 sec latency 102.826 ms 2024-09-21T01:52:43.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 77542 6.55 MB/sec execute 512 sec latency 97.846 ms 2024-09-21T01:52:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:43 smithi090 ceph-mon[35142]: pgmap v948: 129 pgs: 129 active+clean; 436 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-09-21T01:52:44.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:43 smithi045 ceph-mon[28361]: pgmap v948: 129 pgs: 129 active+clean; 436 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.8 MiB/s wr, 176 op/s 2024-09-21T01:52:44.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 350781 29.73 MB/sec execute 511 sec latency 109.193 ms 2024-09-21T01:52:44.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 77696 6.57 MB/sec execute 513 sec latency 103.564 ms 2024-09-21T01:52:45.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 351881 29.78 MB/sec execute 512 sec latency 97.838 ms 2024-09-21T01:52:45.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 77877 6.58 MB/sec execute 514 sec latency 112.352 ms 2024-09-21T01:52:46.242 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:45 smithi090 ceph-mon[35142]: pgmap v949: 129 pgs: 129 active+clean; 437 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.5 MiB/s wr, 126 op/s 2024-09-21T01:52:46.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:45 smithi045 ceph-mon[28361]: pgmap v949: 129 pgs: 129 active+clean; 437 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.5 MiB/s wr, 126 op/s 2024-09-21T01:52:46.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 352093 29.73 MB/sec execute 513 sec latency 100.712 ms 2024-09-21T01:52:46.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78001 6.59 MB/sec execute 515 sec latency 129.927 ms 2024-09-21T01:52:47.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 353314 29.78 MB/sec execute 514 sec latency 202.155 ms 2024-09-21T01:52:47.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78099 6.59 MB/sec execute 516 sec latency 200.220 ms 2024-09-21T01:52:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:47 smithi090 ceph-mon[35142]: pgmap v950: 129 pgs: 129 active+clean; 446 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.6 MiB/s wr, 142 op/s 2024-09-21T01:52:48.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:47 smithi045 ceph-mon[28361]: pgmap v950: 129 pgs: 129 active+clean; 446 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.6 MiB/s wr, 142 op/s 2024-09-21T01:52:48.717 INFO:tasks.workunit.client.0.smithi045.stdout: 10 354390 29.87 MB/sec execute 515 sec latency 107.771 ms 2024-09-21T01:52:48.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78189 6.58 MB/sec execute 517 sec latency 102.276 ms 2024-09-21T01:52:49.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 355441 29.92 MB/sec execute 516 sec latency 100.532 ms 2024-09-21T01:52:49.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78285 6.58 MB/sec execute 518 sec latency 133.425 ms 2024-09-21T01:52:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:49 smithi090 ceph-mon[35142]: pgmap v951: 129 pgs: 129 active+clean; 452 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-09-21T01:52:50.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:49 smithi045 ceph-mon[28361]: pgmap v951: 129 pgs: 129 active+clean; 452 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-09-21T01:52:50.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 356124 29.93 MB/sec execute 517 sec latency 94.690 ms 2024-09-21T01:52:50.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78422 6.58 MB/sec execute 519 sec latency 114.346 ms 2024-09-21T01:52:51.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 357530 30.01 MB/sec execute 518 sec latency 98.697 ms 2024-09-21T01:52:51.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78560 6.59 MB/sec execute 520 sec latency 109.995 ms 2024-09-21T01:52:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:51 smithi090 ceph-mon[35142]: pgmap v952: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.9 MiB/s wr, 123 op/s 2024-09-21T01:52:52.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:51 smithi045 ceph-mon[28361]: pgmap v952: 129 pgs: 129 active+clean; 455 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.9 MiB/s wr, 123 op/s 2024-09-21T01:52:52.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 358648 30.08 MB/sec execute 519 sec latency 367.319 ms 2024-09-21T01:52:52.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78667 6.58 MB/sec execute 521 sec latency 119.587 ms 2024-09-21T01:52:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:53.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:52:53.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 358990 30.06 MB/sec execute 520 sec latency 210.884 ms 2024-09-21T01:52:53.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78785 6.58 MB/sec execute 522 sec latency 121.822 ms 2024-09-21T01:52:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:53 smithi090 ceph-mon[35142]: pgmap v953: 129 pgs: 129 active+clean; 455 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 133 op/s 2024-09-21T01:52:54.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:53 smithi045 ceph-mon[28361]: pgmap v953: 129 pgs: 129 active+clean; 455 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 133 op/s 2024-09-21T01:52:54.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 359560 30.07 MB/sec execute 521 sec latency 95.921 ms 2024-09-21T01:52:54.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 78895 6.57 MB/sec execute 523 sec latency 147.966 ms 2024-09-21T01:52:55.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 360939 30.14 MB/sec execute 522 sec latency 125.242 ms 2024-09-21T01:52:55.957 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79016 6.57 MB/sec execute 524 sec latency 143.160 ms 2024-09-21T01:52:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:55 smithi045 ceph-mon[28361]: pgmap v954: 129 pgs: 129 active+clean; 454 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.4 MiB/s wr, 89 op/s 2024-09-21T01:52:56.242 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:55 smithi090 ceph-mon[35142]: pgmap v954: 129 pgs: 129 active+clean; 454 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.4 MiB/s wr, 89 op/s 2024-09-21T01:52:56.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 362198 30.22 MB/sec execute 523 sec latency 138.053 ms 2024-09-21T01:52:56.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79153 6.56 MB/sec execute 525 sec latency 108.355 ms 2024-09-21T01:52:57.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 362574 30.20 MB/sec execute 524 sec latency 141.919 ms 2024-09-21T01:52:57.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79300 6.57 MB/sec execute 526 sec latency 117.418 ms 2024-09-21T01:52:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:57 smithi090 ceph-mon[35142]: pgmap v955: 129 pgs: 129 active+clean; 451 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.1 MiB/s wr, 130 op/s 2024-09-21T01:52:58.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:57 smithi045 ceph-mon[28361]: pgmap v955: 129 pgs: 129 active+clean; 451 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.1 MiB/s wr, 130 op/s 2024-09-21T01:52:58.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 362586 30.14 MB/sec execute 525 sec latency 473.435 ms 2024-09-21T01:52:58.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79391 6.56 MB/sec execute 527 sec latency 292.585 ms 2024-09-21T01:52:59.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 362644 30.08 MB/sec execute 526 sec latency 483.827 ms 2024-09-21T01:52:59.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79499 6.56 MB/sec execute 528 sec latency 215.457 ms 2024-09-21T01:53:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:52:59 smithi090 ceph-mon[35142]: pgmap v956: 129 pgs: 129 active+clean; 445 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-09-21T01:53:00.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:52:59 smithi045 ceph-mon[28361]: pgmap v956: 129 pgs: 129 active+clean; 445 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.5 MiB/s wr, 155 op/s 2024-09-21T01:53:00.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 362682 30.03 MB/sec execute 527 sec latency 398.871 ms 2024-09-21T01:53:00.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79602 6.55 MB/sec execute 529 sec latency 188.920 ms 2024-09-21T01:53:01.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 362745 29.97 MB/sec execute 528 sec latency 442.515 ms 2024-09-21T01:53:01.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79703 6.55 MB/sec execute 530 sec latency 285.172 ms 2024-09-21T01:53:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:01 smithi090 ceph-mon[35142]: pgmap v957: 129 pgs: 129 active+clean; 421 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.1 MiB/s wr, 137 op/s 2024-09-21T01:53:02.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:01 smithi045 ceph-mon[28361]: pgmap v957: 129 pgs: 129 active+clean; 421 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 3.1 MiB/s wr, 137 op/s 2024-09-21T01:53:02.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 363215 29.98 MB/sec execute 529 sec latency 451.116 ms 2024-09-21T01:53:02.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79800 6.55 MB/sec execute 531 sec latency 295.877 ms 2024-09-21T01:53:03.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 363411 29.92 MB/sec execute 530 sec latency 476.749 ms 2024-09-21T01:53:03.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 79903 6.54 MB/sec execute 532 sec latency 199.135 ms 2024-09-21T01:53:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:03 smithi090 ceph-mon[35142]: pgmap v958: 129 pgs: 129 active+clean; 417 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.0 MiB/s wr, 173 op/s 2024-09-21T01:53:04.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:03 smithi045 ceph-mon[28361]: pgmap v958: 129 pgs: 129 active+clean; 417 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.0 MiB/s wr, 173 op/s 2024-09-21T01:53:04.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 363580 29.89 MB/sec execute 531 sec latency 613.849 ms 2024-09-21T01:53:04.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80012 6.54 MB/sec execute 533 sec latency 175.396 ms 2024-09-21T01:53:05.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 364021 29.86 MB/sec execute 532 sec latency 555.897 ms 2024-09-21T01:53:05.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80156 6.54 MB/sec execute 534 sec latency 175.426 ms 2024-09-21T01:53:06.242 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:05 smithi090 ceph-mon[35142]: pgmap v959: 129 pgs: 129 active+clean; 417 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.5 MiB/s wr, 158 op/s 2024-09-21T01:53:06.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:05 smithi045 ceph-mon[28361]: pgmap v959: 129 pgs: 129 active+clean; 417 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.5 MiB/s wr, 158 op/s 2024-09-21T01:53:06.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 364312 29.85 MB/sec execute 533 sec latency 538.527 ms 2024-09-21T01:53:06.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80276 6.54 MB/sec execute 535 sec latency 177.091 ms 2024-09-21T01:53:07.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 364587 29.83 MB/sec execute 534 sec latency 443.196 ms 2024-09-21T01:53:07.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80404 6.53 MB/sec execute 536 sec latency 196.808 ms 2024-09-21T01:53:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:07 smithi090 ceph-mon[35142]: pgmap v960: 129 pgs: 129 active+clean; 417 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-09-21T01:53:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:08.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:07 smithi045 ceph-mon[28361]: pgmap v960: 129 pgs: 129 active+clean; 417 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-09-21T01:53:08.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:08.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 364888 29.81 MB/sec execute 535 sec latency 507.802 ms 2024-09-21T01:53:08.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80529 6.53 MB/sec execute 537 sec latency 142.427 ms 2024-09-21T01:53:09.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 365005 29.77 MB/sec execute 536 sec latency 677.144 ms 2024-09-21T01:53:09.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80631 6.53 MB/sec execute 538 sec latency 211.679 ms 2024-09-21T01:53:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:09 smithi090 ceph-mon[35142]: pgmap v961: 129 pgs: 129 active+clean; 421 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.2 MiB/s wr, 161 op/s 2024-09-21T01:53:10.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:09 smithi045 ceph-mon[28361]: pgmap v961: 129 pgs: 129 active+clean; 421 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 2.2 MiB/s wr, 161 op/s 2024-09-21T01:53:10.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 365598 29.77 MB/sec execute 537 sec latency 472.931 ms 2024-09-21T01:53:10.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80759 6.54 MB/sec execute 539 sec latency 182.758 ms 2024-09-21T01:53:11.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 365747 29.73 MB/sec execute 538 sec latency 503.992 ms 2024-09-21T01:53:11.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 80861 6.54 MB/sec execute 540 sec latency 284.354 ms 2024-09-21T01:53:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:11 smithi090 ceph-mon[35142]: pgmap v962: 129 pgs: 129 active+clean; 421 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:53:12.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:11 smithi045 ceph-mon[28361]: pgmap v962: 129 pgs: 129 active+clean; 421 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.8 MiB/s wr, 125 op/s 2024-09-21T01:53:12.718 INFO:tasks.workunit.client.0.smithi045.stdout: 10 365935 29.70 MB/sec execute 539 sec latency 449.566 ms 2024-09-21T01:53:12.958 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81017 6.54 MB/sec execute 541 sec latency 183.192 ms 2024-09-21T01:53:13.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 366060 29.65 MB/sec execute 540 sec latency 492.350 ms 2024-09-21T01:53:13.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81089 6.54 MB/sec execute 542 sec latency 173.026 ms 2024-09-21T01:53:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:13 smithi090 ceph-mon[35142]: pgmap v963: 129 pgs: 129 active+clean; 428 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-09-21T01:53:14.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:13 smithi045 ceph-mon[28361]: pgmap v963: 129 pgs: 129 active+clean; 428 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.5 MiB/s wr, 147 op/s 2024-09-21T01:53:14.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 366070 29.60 MB/sec execute 541 sec latency 399.247 ms 2024-09-21T01:53:14.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81223 6.55 MB/sec execute 543 sec latency 222.214 ms 2024-09-21T01:53:15.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 366224 29.55 MB/sec execute 542 sec latency 435.029 ms 2024-09-21T01:53:15.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81406 6.57 MB/sec execute 544 sec latency 98.869 ms 2024-09-21T01:53:16.243 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:15 smithi090 ceph-mon[35142]: pgmap v964: 129 pgs: 129 active+clean; 430 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.1 MiB/s wr, 100 op/s 2024-09-21T01:53:16.377 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:15 smithi045 ceph-mon[28361]: pgmap v964: 129 pgs: 129 active+clean; 430 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.1 MiB/s wr, 100 op/s 2024-09-21T01:53:16.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 367492 29.60 MB/sec execute 543 sec latency 103.743 ms 2024-09-21T01:53:16.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81543 6.58 MB/sec execute 545 sec latency 103.343 ms 2024-09-21T01:53:17.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 368563 29.68 MB/sec execute 544 sec latency 133.566 ms 2024-09-21T01:53:17.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81656 6.58 MB/sec execute 546 sec latency 115.395 ms 2024-09-21T01:53:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:17 smithi090 ceph-mon[35142]: pgmap v965: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.9 MiB/s wr, 110 op/s 2024-09-21T01:53:18.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:17 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:53:18.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:17 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:18.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:17 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:18.419 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:17 smithi045 ceph-mon[28361]: pgmap v965: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.9 MiB/s wr, 110 op/s 2024-09-21T01:53:18.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:17 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:53:18.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:17 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:18.420 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:17 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:18.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 369607 29.73 MB/sec execute 545 sec latency 174.553 ms 2024-09-21T01:53:18.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81731 6.57 MB/sec execute 547 sec latency 138.743 ms 2024-09-21T01:53:19.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 369687 29.68 MB/sec execute 546 sec latency 419.344 ms 2024-09-21T01:53:19.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: pgmap v966: 129 pgs: 129 active+clean; 441 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.3 MiB/s wr, 121 op/s 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:53:19.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81805 6.57 MB/sec execute 548 sec latency 173.845 ms 2024-09-21T01:53:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: pgmap v966: 129 pgs: 129 active+clean; 441 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.3 MiB/s wr, 121 op/s 2024-09-21T01:53:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:53:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:53:19.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:53:20.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 369747 29.63 MB/sec execute 547 sec latency 529.195 ms 2024-09-21T01:53:20.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81893 6.57 MB/sec execute 549 sec latency 171.291 ms 2024-09-21T01:53:21.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 369827 29.58 MB/sec execute 548 sec latency 400.329 ms 2024-09-21T01:53:21.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:21 smithi045 ceph-mon[28361]: pgmap v967: 129 pgs: 129 active+clean; 443 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.4 MiB/s wr, 94 op/s 2024-09-21T01:53:21.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 81968 6.56 MB/sec execute 550 sec latency 247.379 ms 2024-09-21T01:53:21.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:21 smithi090 ceph-mon[35142]: pgmap v967: 129 pgs: 129 active+clean; 443 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.4 MiB/s wr, 94 op/s 2024-09-21T01:53:22.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 370288 29.58 MB/sec execute 549 sec latency 463.764 ms 2024-09-21T01:53:22.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:22.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82070 6.56 MB/sec execute 551 sec latency 205.319 ms 2024-09-21T01:53:22.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:23.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 370473 29.53 MB/sec execute 550 sec latency 518.353 ms 2024-09-21T01:53:23.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:23 smithi045 ceph-mon[28361]: pgmap v968: 129 pgs: 129 active+clean; 447 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 122 op/s 2024-09-21T01:53:23.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82149 6.55 MB/sec execute 552 sec latency 225.922 ms 2024-09-21T01:53:23.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:23 smithi090 ceph-mon[35142]: pgmap v968: 129 pgs: 129 active+clean; 447 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.0 MiB/s wr, 122 op/s 2024-09-21T01:53:24.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 370677 29.50 MB/sec execute 551 sec latency 409.469 ms 2024-09-21T01:53:24.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82234 6.56 MB/sec execute 553 sec latency 217.564 ms 2024-09-21T01:53:25.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 371114 29.47 MB/sec execute 552 sec latency 632.559 ms 2024-09-21T01:53:25.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:25 smithi045 ceph-mon[28361]: pgmap v969: 129 pgs: 129 active+clean; 446 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.2 MiB/s wr, 90 op/s 2024-09-21T01:53:25.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82308 6.55 MB/sec execute 554 sec latency 323.728 ms 2024-09-21T01:53:25.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:25 smithi090 ceph-mon[35142]: pgmap v969: 129 pgs: 129 active+clean; 446 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.2 MiB/s wr, 90 op/s 2024-09-21T01:53:26.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 371405 29.46 MB/sec execute 553 sec latency 519.829 ms 2024-09-21T01:53:26.959 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82385 6.54 MB/sec execute 555 sec latency 253.591 ms 2024-09-21T01:53:27.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 372147 29.50 MB/sec execute 554 sec latency 383.268 ms 2024-09-21T01:53:27.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:27 smithi045 ceph-mon[28361]: pgmap v970: 129 pgs: 129 active+clean; 446 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.6 MiB/s wr, 98 op/s 2024-09-21T01:53:27.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82518 6.54 MB/sec execute 556 sec latency 280.889 ms 2024-09-21T01:53:27.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:27 smithi090 ceph-mon[35142]: pgmap v970: 129 pgs: 129 active+clean; 446 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.6 MiB/s wr, 98 op/s 2024-09-21T01:53:28.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 373206 29.54 MB/sec execute 555 sec latency 111.432 ms 2024-09-21T01:53:28.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82659 6.53 MB/sec execute 557 sec latency 104.688 ms 2024-09-21T01:53:29.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 373840 29.55 MB/sec execute 556 sec latency 101.346 ms 2024-09-21T01:53:29.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:29 smithi045 ceph-mon[28361]: pgmap v971: 129 pgs: 129 active+clean; 445 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.1 MiB/s wr, 116 op/s 2024-09-21T01:53:29.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82814 6.53 MB/sec execute 558 sec latency 114.203 ms 2024-09-21T01:53:29.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:29 smithi090 ceph-mon[35142]: pgmap v971: 129 pgs: 129 active+clean; 445 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.1 MiB/s wr, 116 op/s 2024-09-21T01:53:30.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 375259 29.63 MB/sec execute 557 sec latency 110.853 ms 2024-09-21T01:53:30.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 82968 6.53 MB/sec execute 559 sec latency 148.302 ms 2024-09-21T01:53:31.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 376477 29.70 MB/sec execute 558 sec latency 209.160 ms 2024-09-21T01:53:31.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:31 smithi045 ceph-mon[28361]: pgmap v972: 129 pgs: 129 active+clean; 445 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 115 op/s 2024-09-21T01:53:31.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 83100 6.53 MB/sec execute 560 sec latency 141.809 ms 2024-09-21T01:53:31.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:31 smithi090 ceph-mon[35142]: pgmap v972: 129 pgs: 129 active+clean; 445 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 115 op/s 2024-09-21T01:53:32.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 376783 29.67 MB/sec execute 559 sec latency 92.067 ms 2024-09-21T01:53:32.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 83266 6.53 MB/sec execute 561 sec latency 110.426 ms 2024-09-21T01:53:33.719 INFO:tasks.workunit.client.0.smithi045.stdout: 10 377671 29.69 MB/sec execute 560 sec latency 191.890 ms 2024-09-21T01:53:33.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:33 smithi045 ceph-mon[28361]: pgmap v973: 129 pgs: 129 active+clean; 437 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.8 MiB/s wr, 139 op/s 2024-09-21T01:53:33.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 83424 6.53 MB/sec execute 562 sec latency 112.350 ms 2024-09-21T01:53:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:33 smithi090 ceph-mon[35142]: pgmap v973: 129 pgs: 129 active+clean; 437 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 2.8 MiB/s wr, 139 op/s 2024-09-21T01:53:34.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 379017 29.78 MB/sec execute 561 sec latency 196.348 ms 2024-09-21T01:53:34.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 83583 6.54 MB/sec execute 563 sec latency 97.061 ms 2024-09-21T01:53:35.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 380274 29.85 MB/sec execute 562 sec latency 129.464 ms 2024-09-21T01:53:35.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:35 smithi045 ceph-mon[28361]: pgmap v974: 129 pgs: 129 active+clean; 436 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.1 MiB/s wr, 109 op/s 2024-09-21T01:53:35.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 83757 6.53 MB/sec execute 564 sec latency 90.851 ms 2024-09-21T01:53:35.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:35 smithi090 ceph-mon[35142]: pgmap v974: 129 pgs: 129 active+clean; 436 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.1 MiB/s wr, 109 op/s 2024-09-21T01:53:36.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 380424 29.81 MB/sec execute 563 sec latency 109.106 ms 2024-09-21T01:53:36.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 83925 6.54 MB/sec execute 565 sec latency 104.601 ms 2024-09-21T01:53:37.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 381561 29.85 MB/sec execute 564 sec latency 102.051 ms 2024-09-21T01:53:37.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:37 smithi045 ceph-mon[28361]: pgmap v975: 129 pgs: 129 active+clean; 434 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-21T01:53:37.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:37.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84057 6.53 MB/sec execute 566 sec latency 93.482 ms 2024-09-21T01:53:37.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:37 smithi090 ceph-mon[35142]: pgmap v975: 129 pgs: 129 active+clean; 434 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-21T01:53:37.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:38.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 382778 29.93 MB/sec execute 565 sec latency 111.234 ms 2024-09-21T01:53:38.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84214 6.54 MB/sec execute 567 sec latency 126.525 ms 2024-09-21T01:53:39.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 383827 29.98 MB/sec execute 566 sec latency 109.915 ms 2024-09-21T01:53:39.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84385 6.55 MB/sec execute 568 sec latency 90.183 ms 2024-09-21T01:53:40.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:39 smithi045 ceph-mon[28361]: pgmap v976: 129 pgs: 129 active+clean; 437 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 2.7 MiB/s wr, 186 op/s 2024-09-21T01:53:40.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:39 smithi090 ceph-mon[35142]: pgmap v976: 129 pgs: 129 active+clean; 437 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 2.7 MiB/s wr, 186 op/s 2024-09-21T01:53:40.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 384003 29.93 MB/sec execute 567 sec latency 177.302 ms 2024-09-21T01:53:40.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84532 6.55 MB/sec execute 569 sec latency 123.819 ms 2024-09-21T01:53:41.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 384891 29.96 MB/sec execute 568 sec latency 289.167 ms 2024-09-21T01:53:41.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84653 6.56 MB/sec execute 570 sec latency 120.335 ms 2024-09-21T01:53:42.153 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:41 smithi045 ceph-mon[28361]: pgmap v977: 129 pgs: 129 active+clean; 439 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.7 MiB/s wr, 165 op/s 2024-09-21T01:53:42.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:41 smithi090 ceph-mon[35142]: pgmap v977: 129 pgs: 129 active+clean; 439 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.7 MiB/s wr, 165 op/s 2024-09-21T01:53:42.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 385867 30.01 MB/sec execute 569 sec latency 439.124 ms 2024-09-21T01:53:42.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84796 6.57 MB/sec execute 571 sec latency 159.971 ms 2024-09-21T01:53:43.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 387088 30.08 MB/sec execute 570 sec latency 456.776 ms 2024-09-21T01:53:43.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 84960 6.58 MB/sec execute 572 sec latency 109.268 ms 2024-09-21T01:53:44.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:43 smithi045 ceph-mon[28361]: pgmap v978: 129 pgs: 129 active+clean; 445 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.0 MiB/s wr, 159 op/s 2024-09-21T01:53:44.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:43 smithi090 ceph-mon[35142]: pgmap v978: 129 pgs: 129 active+clean; 445 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.0 MiB/s wr, 159 op/s 2024-09-21T01:53:44.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 387471 30.06 MB/sec execute 571 sec latency 112.569 ms 2024-09-21T01:53:44.961 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85071 6.58 MB/sec execute 573 sec latency 114.583 ms 2024-09-21T01:53:45.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 388107 30.06 MB/sec execute 572 sec latency 235.455 ms 2024-09-21T01:53:45.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85196 6.59 MB/sec execute 574 sec latency 151.703 ms 2024-09-21T01:53:46.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:45 smithi045 ceph-mon[28361]: pgmap v979: 129 pgs: 129 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-09-21T01:53:46.244 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:45 smithi090 ceph-mon[35142]: pgmap v979: 129 pgs: 129 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.0 MiB/s wr, 150 op/s 2024-09-21T01:53:46.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 389139 30.09 MB/sec execute 573 sec latency 330.606 ms 2024-09-21T01:53:46.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85272 6.58 MB/sec execute 575 sec latency 149.817 ms 2024-09-21T01:53:47.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 390442 30.18 MB/sec execute 574 sec latency 475.218 ms 2024-09-21T01:53:47.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85361 6.58 MB/sec execute 576 sec latency 109.905 ms 2024-09-21T01:53:48.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:47 smithi045 ceph-mon[28361]: pgmap v980: 129 pgs: 129 active+clean; 456 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.0 MiB/s wr, 158 op/s 2024-09-21T01:53:48.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:47 smithi090 ceph-mon[35142]: pgmap v980: 129 pgs: 129 active+clean; 456 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.0 MiB/s wr, 158 op/s 2024-09-21T01:53:48.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 390903 30.18 MB/sec execute 575 sec latency 511.067 ms 2024-09-21T01:53:48.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85463 6.58 MB/sec execute 577 sec latency 153.050 ms 2024-09-21T01:53:49.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 390937 30.12 MB/sec execute 576 sec latency 485.778 ms 2024-09-21T01:53:49.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85545 6.57 MB/sec execute 578 sec latency 177.446 ms 2024-09-21T01:53:50.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:49 smithi045 ceph-mon[28361]: pgmap v981: 129 pgs: 129 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 136 op/s 2024-09-21T01:53:50.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:49 smithi090 ceph-mon[35142]: pgmap v981: 129 pgs: 129 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 136 op/s 2024-09-21T01:53:50.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 390945 30.07 MB/sec execute 577 sec latency 434.009 ms 2024-09-21T01:53:50.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85643 6.58 MB/sec execute 579 sec latency 197.365 ms 2024-09-21T01:53:51.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 390989 30.02 MB/sec execute 578 sec latency 425.411 ms 2024-09-21T01:53:51.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85739 6.58 MB/sec execute 580 sec latency 249.337 ms 2024-09-21T01:53:52.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:51 smithi045 ceph-mon[28361]: pgmap v982: 129 pgs: 129 active+clean; 445 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.7 MiB/s wr, 104 op/s 2024-09-21T01:53:52.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:51 smithi090 ceph-mon[35142]: pgmap v982: 129 pgs: 129 active+clean; 445 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.7 MiB/s wr, 104 op/s 2024-09-21T01:53:52.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 391006 29.97 MB/sec execute 579 sec latency 451.032 ms 2024-09-21T01:53:52.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85848 6.57 MB/sec execute 581 sec latency 136.018 ms 2024-09-21T01:53:53.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:52 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:53.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:52 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:53:53.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 391028 29.92 MB/sec execute 580 sec latency 408.873 ms 2024-09-21T01:53:53.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85916 6.56 MB/sec execute 582 sec latency 282.619 ms 2024-09-21T01:53:54.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:53 smithi045 ceph-mon[28361]: pgmap v983: 129 pgs: 129 active+clean; 436 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.0 MiB/s wr, 123 op/s 2024-09-21T01:53:54.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:53 smithi090 ceph-mon[35142]: pgmap v983: 129 pgs: 129 active+clean; 436 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.0 MiB/s wr, 123 op/s 2024-09-21T01:53:54.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 391098 29.87 MB/sec execute 581 sec latency 375.899 ms 2024-09-21T01:53:54.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 85998 6.56 MB/sec execute 583 sec latency 191.835 ms 2024-09-21T01:53:55.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 391150 29.82 MB/sec execute 582 sec latency 434.179 ms 2024-09-21T01:53:55.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86099 6.55 MB/sec execute 584 sec latency 192.744 ms 2024-09-21T01:53:56.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:55 smithi045 ceph-mon[28361]: pgmap v984: 129 pgs: 129 active+clean; 435 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.3 MiB/s wr, 102 op/s 2024-09-21T01:53:56.244 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:55 smithi090 ceph-mon[35142]: pgmap v984: 129 pgs: 129 active+clean; 435 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.3 MiB/s wr, 102 op/s 2024-09-21T01:53:56.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 391541 29.82 MB/sec execute 583 sec latency 479.444 ms 2024-09-21T01:53:56.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86181 6.54 MB/sec execute 585 sec latency 298.535 ms 2024-09-21T01:53:57.720 INFO:tasks.workunit.client.0.smithi045.stdout: 10 391890 29.78 MB/sec execute 584 sec latency 395.851 ms 2024-09-21T01:53:57.919 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:57 smithi045 ceph-mon[28361]: pgmap v985: 129 pgs: 129 active+clean; 434 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.6 MiB/s wr, 120 op/s 2024-09-21T01:53:57.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86282 6.53 MB/sec execute 586 sec latency 197.162 ms 2024-09-21T01:53:58.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:57 smithi090 ceph-mon[35142]: pgmap v985: 129 pgs: 129 active+clean; 434 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.6 MiB/s wr, 120 op/s 2024-09-21T01:53:58.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 392038 29.75 MB/sec execute 585 sec latency 443.877 ms 2024-09-21T01:53:58.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86401 6.54 MB/sec execute 587 sec latency 171.593 ms 2024-09-21T01:53:59.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 392471 29.72 MB/sec execute 586 sec latency 525.937 ms 2024-09-21T01:53:59.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86487 6.53 MB/sec execute 588 sec latency 163.966 ms 2024-09-21T01:54:00.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:53:59 smithi045 ceph-mon[28361]: pgmap v986: 129 pgs: 129 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.7 MiB/s wr, 118 op/s 2024-09-21T01:54:00.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:53:59 smithi090 ceph-mon[35142]: pgmap v986: 129 pgs: 129 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.7 MiB/s wr, 118 op/s 2024-09-21T01:54:00.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 392859 29.72 MB/sec execute 587 sec latency 544.349 ms 2024-09-21T01:54:00.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86603 6.53 MB/sec execute 589 sec latency 169.978 ms 2024-09-21T01:54:01.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 393044 29.69 MB/sec execute 588 sec latency 558.176 ms 2024-09-21T01:54:01.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86724 6.52 MB/sec execute 590 sec latency 168.312 ms 2024-09-21T01:54:02.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:01 smithi045 ceph-mon[28361]: pgmap v987: 129 pgs: 129 active+clean; 426 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-09-21T01:54:02.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:01 smithi090 ceph-mon[35142]: pgmap v987: 129 pgs: 129 active+clean; 426 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 2.4 MiB/s wr, 107 op/s 2024-09-21T01:54:02.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 393403 29.69 MB/sec execute 589 sec latency 349.841 ms 2024-09-21T01:54:02.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86841 6.53 MB/sec execute 591 sec latency 179.726 ms 2024-09-21T01:54:03.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 393471 29.64 MB/sec execute 590 sec latency 595.164 ms 2024-09-21T01:54:03.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 86979 6.52 MB/sec execute 592 sec latency 111.398 ms 2024-09-21T01:54:04.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:03 smithi045 ceph-mon[28361]: pgmap v988: 129 pgs: 129 active+clean; 422 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.5 MiB/s wr, 149 op/s 2024-09-21T01:54:04.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:03 smithi090 ceph-mon[35142]: pgmap v988: 129 pgs: 129 active+clean; 422 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.5 MiB/s wr, 149 op/s 2024-09-21T01:54:04.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 394360 29.67 MB/sec execute 591 sec latency 376.561 ms 2024-09-21T01:54:04.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 87134 6.53 MB/sec execute 593 sec latency 145.495 ms 2024-09-21T01:54:05.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 394575 29.64 MB/sec execute 592 sec latency 155.493 ms 2024-09-21T01:54:05.962 INFO:tasks.workunit.client.1.smithi090.stdout: 10 87324 6.53 MB/sec execute 594 sec latency 103.076 ms 2024-09-21T01:54:06.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:05 smithi045 ceph-mon[28361]: pgmap v989: 129 pgs: 129 active+clean; 422 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 MiB/s wr, 124 op/s 2024-09-21T01:54:06.245 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:05 smithi090 ceph-mon[35142]: pgmap v989: 129 pgs: 129 active+clean; 422 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 MiB/s wr, 124 op/s 2024-09-21T01:54:06.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 395438 29.65 MB/sec execute 593 sec latency 159.978 ms 2024-09-21T01:54:06.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 87459 6.53 MB/sec execute 595 sec latency 102.144 ms 2024-09-21T01:54:07.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 396716 29.73 MB/sec execute 594 sec latency 138.164 ms 2024-09-21T01:54:07.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 87602 6.53 MB/sec execute 596 sec latency 102.483 ms 2024-09-21T01:54:08.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:07 smithi045 ceph-mon[28361]: pgmap v990: 129 pgs: 129 active+clean; 425 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.5 MiB/s wr, 168 op/s 2024-09-21T01:54:08.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:07 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:07 smithi090 ceph-mon[35142]: pgmap v990: 129 pgs: 129 active+clean; 425 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.5 MiB/s wr, 168 op/s 2024-09-21T01:54:08.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:07 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:08.721 INFO:tasks.workunit.client.0.smithi045.stdout: 10 397725 29.78 MB/sec execute 595 sec latency 277.439 ms 2024-09-21T01:54:08.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 87771 6.53 MB/sec execute 597 sec latency 93.114 ms 2024-09-21T01:54:09.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 398039 29.76 MB/sec execute 596 sec latency 325.421 ms 2024-09-21T01:54:09.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 87915 6.54 MB/sec execute 598 sec latency 162.982 ms 2024-09-21T01:54:10.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:09 smithi045 ceph-mon[28361]: pgmap v991: 129 pgs: 129 active+clean; 427 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.1 MiB/s wr, 145 op/s 2024-09-21T01:54:10.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:09 smithi090 ceph-mon[35142]: pgmap v991: 129 pgs: 129 active+clean; 427 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.1 MiB/s wr, 145 op/s 2024-09-21T01:54:10.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 398090 29.71 MB/sec execute 597 sec latency 442.498 ms 2024-09-21T01:54:10.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 88015 6.54 MB/sec execute 599 sec latency 309.452 ms 2024-09-21T01:54:11.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 398095 29.66 MB/sec execute 598 sec latency 427.610 ms 2024-09-21T01:54:11.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 cleanup 600 sec 2024-09-21T01:54:12.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:11 smithi045 ceph-mon[28361]: pgmap v992: 129 pgs: 129 active+clean; 431 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.5 MiB/s wr, 162 op/s 2024-09-21T01:54:12.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:11 smithi090 ceph-mon[35142]: pgmap v992: 129 pgs: 129 active+clean; 431 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.5 MiB/s wr, 162 op/s 2024-09-21T01:54:12.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 398118 29.61 MB/sec execute 599 sec latency 375.665 ms 2024-09-21T01:54:12.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 cleanup 601 sec 2024-09-21T01:54:13.722 INFO:tasks.workunit.client.0.smithi045.stdout: 10 cleanup 600 sec 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: 0 cleanup 600 sec 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: Operation Count AvgLat MaxLat 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: ---------------------------------------- 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: NTCreateX 568946 0.064 144.664 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: Close 417746 0.001 0.101 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: Rename 24096 0.392 122.277 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: Unlink 115152 0.603 147.074 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: Qpathinfo 515996 0.037 118.156 2024-09-21T01:54:13.915 INFO:tasks.workunit.client.0.smithi045.stdout: Qfileinfo 89943 0.001 0.071 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: Qfsinfo 94674 0.002 0.047 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: Sfileinfo 46264 1.957 162.262 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: Find 199461 0.023 7.094 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: WriteX 281072 0.463 180.640 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: ReadX 893138 0.026 171.316 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: LockX 1860 0.003 0.024 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: UnlockX 1860 0.002 0.028 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: Flush 39904 140.668 715.073 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout: 2024-09-21T01:54:13.916 INFO:tasks.workunit.client.0.smithi045.stdout:Throughput 29.6094 MB/sec 10 clients 10 procs max_latency=715.084 ms 2024-09-21T01:54:13.916 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T01:54:13.917 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T01:54:13.963 INFO:tasks.workunit.client.1.smithi090.stdout: 10 cleanup 602 sec 2024-09-21T01:54:14.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:13 smithi045 ceph-mon[28361]: pgmap v993: 129 pgs: 129 active+clean; 438 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.1 MiB/s wr, 163 op/s 2024-09-21T01:54:14.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:13 smithi090 ceph-mon[35142]: pgmap v993: 129 pgs: 129 active+clean; 438 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.1 MiB/s wr, 163 op/s 2024-09-21T01:54:14.428 INFO:tasks.workunit.client.1.smithi090.stdout: 0 cleanup 602 sec 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Operation Count AvgLat MaxLat 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: ---------------------------------------- 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: NTCreateX 125911 10.287 305.804 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Close 92806 0.111 9.661 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Rename 5337 26.060 205.432 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Unlink 25131 9.120 170.806 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Qpathinfo 114527 5.774 211.629 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Qfileinfo 20051 0.202 5.407 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Qfsinfo 20748 0.573 71.046 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Sfileinfo 10283 1.495 94.276 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: Find 44064 79.406 358.790 2024-09-21T01:54:14.429 INFO:tasks.workunit.client.1.smithi090.stdout: WriteX 62301 0.435 100.929 2024-09-21T01:54:14.430 INFO:tasks.workunit.client.1.smithi090.stdout: ReadX 197866 0.006 2.534 2024-09-21T01:54:14.430 INFO:tasks.workunit.client.1.smithi090.stdout: LockX 410 4.044 68.785 2024-09-21T01:54:14.430 INFO:tasks.workunit.client.1.smithi090.stdout: UnlockX 410 3.645 61.438 2024-09-21T01:54:14.430 INFO:tasks.workunit.client.1.smithi090.stdout: Flush 8732 11.641 286.085 2024-09-21T01:54:14.430 INFO:tasks.workunit.client.1.smithi090.stdout: 2024-09-21T01:54:14.430 INFO:tasks.workunit.client.1.smithi090.stdout:Throughput 6.5407 MB/sec 10 clients 10 procs max_latency=358.796 ms 2024-09-21T01:54:14.431 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T01:54:14.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-09-21T01:54:15.093 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-09-21T01:54:15.093 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T01:54:16.029 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.1... 2024-09-21T01:54:16.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-09-21T01:54:16.063 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:16.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:15 smithi045 ceph-mon[28361]: pgmap v994: 129 pgs: 129 active+clean; 438 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-09-21T01:54:16.245 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:15 smithi090 ceph-mon[35142]: pgmap v994: 129 pgs: 129 active+clean; 438 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.7 MiB/s wr, 117 op/s 2024-09-21T01:54:16.360 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:16.360 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T01:54:16.387 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T01:54:16.387 DEBUG:teuthology.orchestra.run.smithi045:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T01:54:16.454 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T01:54:16.455 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-09-21T01:54:16.520 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-09-21T01:54:16.520 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T01:54:17.104 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T01:54:17.104 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi045.front.sepia.ceph.com 2024-09-21T01:54:17.104 DEBUG:teuthology.orchestra.run.smithi045:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T01:54:17.104 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-09-21T01:54:17.105 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T01:54:18.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:17 smithi045 ceph-mon[28361]: pgmap v995: 129 pgs: 129 active+clean; 383 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.3 MiB/s wr, 155 op/s 2024-09-21T01:54:18.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:17 smithi090 ceph-mon[35142]: pgmap v995: 129 pgs: 129 active+clean; 383 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.3 MiB/s wr, 155 op/s 2024-09-21T01:54:18.980 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:54:18.981 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:54:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-09-21T01:54:18.981 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:54:18.981 INFO:teuthology.orchestra.run.smithi045.stdout:Removing: 2024-09-21T01:54:18.981 INFO:teuthology.orchestra.run.smithi045.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T01:54:18.981 INFO:teuthology.orchestra.run.smithi045.stdout:Removing unused dependencies: 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout:Remove 2 Packages 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout:Freed space: 5.5 M 2024-09-21T01:54:18.982 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:54:19.041 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:54:19.041 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:54:19.183 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout:Removing: 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout:Removing unused dependencies: 2024-09-21T01:54:19.184 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout:Remove 2 Packages 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout:Freed space: 5.5 M 2024-09-21T01:54:19.185 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:54:19.254 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:54:19.254 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:54:19.300 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:54:19.300 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:54:19.518 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:54:19.563 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:54:19.563 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:54:19.660 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T01:54:19.769 INFO:teuthology.orchestra.run.smithi045.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T01:54:19.917 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:54:20.139 INFO:teuthology.orchestra.run.smithi090.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T01:54:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:19 smithi045 ceph-mon[28361]: pgmap v996: 129 pgs: 129 active+clean; 316 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 2.8 MiB/s wr, 134 op/s 2024-09-21T01:54:20.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:54:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:54:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:54:20.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:19 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:54:20.224 INFO:teuthology.orchestra.run.smithi090.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T01:54:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:19 smithi090 ceph-mon[35142]: pgmap v996: 129 pgs: 129 active+clean; 316 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 2.8 MiB/s wr, 134 op/s 2024-09-21T01:54:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T01:54:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T01:54:20.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T01:54:20.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:19 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' 2024-09-21T01:54:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T01:54:20.650 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T01:54:21.190 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T01:54:21.190 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:21.190 INFO:teuthology.orchestra.run.smithi045.stdout:Removed: 2024-09-21T01:54:21.191 INFO:teuthology.orchestra.run.smithi045.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T01:54:21.191 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:21.191 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:54:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T01:54:21.235 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T01:54:21.831 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T01:54:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:21.832 INFO:teuthology.orchestra.run.smithi090.stdout:Removed: 2024-09-21T01:54:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T01:54:21.832 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:21.832 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:54:22.144 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:21 smithi090 ceph-mon[35142]: pgmap v997: 129 pgs: 129 active+clean; 274 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.5 MiB/s wr, 137 op/s 2024-09-21T01:54:22.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:21 smithi045 ceph-mon[28361]: pgmap v997: 129 pgs: 129 active+clean; 274 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.5 MiB/s wr, 137 op/s 2024-09-21T01:54:22.217 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:36:34 ago on Sat 21 Sep 2024 01:17:48 AM UTC. 2024-09-21T01:54:22.360 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:22.361 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-09-21T01:54:22.362 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-09-21T01:54:22.362 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-09-21T01:54:22.362 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:22.373 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 2.0 M 2024-09-21T01:54:22.373 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 9.2 M 2024-09-21T01:54:22.373 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-09-21T01:54:22.652 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 251 kB/s | 44 kB 00:00 2024-09-21T01:54:22.678 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 525 kB/s | 106 kB 00:00 2024-09-21T01:54:22.736 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 860 kB/s | 72 kB 00:00 2024-09-21T01:54:22.761 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.0 MB/s | 84 kB 00:00 2024-09-21T01:54:22.803 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.7 MB/s | 906 kB 00:00 2024-09-21T01:54:22.854 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.0 MB/s | 837 kB 00:00 2024-09-21T01:54:22.854 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:54:22.854 INFO:teuthology.orchestra.run.smithi045.stdout:Total 4.2 MB/s | 2.0 MB 00:00 2024-09-21T01:54:22.885 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:36:27 ago on Sat 21 Sep 2024 01:17:55 AM UTC. 2024-09-21T01:54:22.940 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-09-21T01:54:22.979 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-09-21T01:54:22.979 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-09-21T01:54:23.024 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-21T01:54:23.025 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:54:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-21T01:54:23.025 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:54:23.025 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-09-21T01:54:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:23.048 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-09-21T01:54:23.048 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 9.2 M 2024-09-21T01:54:23.048 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-21T01:54:23.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:22 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:23.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:22 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:23.431 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 648 kB/s | 44 kB 00:00 2024-09-21T01:54:23.465 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 1.0 MB/s | 106 kB 00:00 2024-09-21T01:54:23.501 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-09-21T01:54:23.501 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-09-21T01:54:23.541 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 5.0 MB/s | 906 kB 00:00 2024-09-21T01:54:23.575 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 499 kB/s | 72 kB 00:00 2024-09-21T01:54:23.609 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 586 kB/s | 84 kB 00:00 2024-09-21T01:54:23.701 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.1 MB/s | 837 kB 00:00 2024-09-21T01:54:23.701 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-21T01:54:23.701 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.1 MB/s | 2.0 MB 00:00 2024-09-21T01:54:23.792 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-21T01:54:23.828 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-21T01:54:23.828 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-21T01:54:24.044 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-09-21T01:54:24.169 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:23 smithi045 ceph-mon[28361]: pgmap v998: 129 pgs: 129 active+clean; 237 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-09-21T01:54:24.183 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-21T01:54:24.233 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-21T01:54:24.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:23 smithi090 ceph-mon[35142]: pgmap v998: 129 pgs: 129 active+clean; 237 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 1.9 MiB/s wr, 122 op/s 2024-09-21T01:54:24.270 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-21T01:54:24.272 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-21T01:54:24.272 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-21T01:54:24.504 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-21T01:54:24.551 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T01:54:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-21T01:54:24.905 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-21T01:54:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-21T01:54:25.001 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-21T01:54:25.187 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T01:54:25.212 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-21T01:54:25.220 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T01:54:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T01:54:25.867 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-21T01:54:25.911 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T01:54:26.165 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:25 smithi090 ceph-mon[35142]: pgmap v999: 129 pgs: 129 active+clean; 237 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-09-21T01:54:26.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:25 smithi045 ceph-mon[28361]: pgmap v999: 129 pgs: 129 active+clean; 237 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 1.0 MiB/s wr, 101 op/s 2024-09-21T01:54:26.585 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T01:54:26.585 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-21T01:54:26.585 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-21T01:54:26.586 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-21T01:54:26.586 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-21T01:54:26.586 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-21T01:54:26.988 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-09-21T01:54:26.989 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-09-21T01:54:27.426 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:27.469 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-21T01:54:27.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-21T01:54:27.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-21T01:54:27.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-21T01:54:27.469 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-21T01:54:27.470 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-21T01:54:27.844 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-21T01:54:28.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:27 smithi045 ceph-mon[28361]: pgmap v1000: 129 pgs: 129 active+clean; 231 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 957 KiB/s wr, 100 op/s 2024-09-21T01:54:28.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:27 smithi090 ceph-mon[35142]: pgmap v1000: 129 pgs: 129 active+clean; 231 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 957 KiB/s wr, 100 op/s 2024-09-21T01:54:28.313 DEBUG:teuthology.parallel:result is None 2024-09-21T01:54:28.314 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-21T01:54:28.325 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-21T01:54:28.326 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:54:28.326 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:54:28.348 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:54:28.348 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:54:28.405 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:54:28.697 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 -- 172.21.15.45:0/2572097729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b01091b0 msgr2=0x7f44b01095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 --2- 172.21.15.45:0/2572097729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b01091b0 0x7f44b01095c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009880 tx=0x7f44a402a4a0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 -- 172.21.15.45:0/2572097729 shutdown_connections 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 --2- 172.21.15.45:0/2572097729 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b01091b0 0x7f44b01095c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 --2- 172.21.15.45:0/2572097729 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44b0103870 0x7f44b0103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 -- 172.21.15.45:0/2572097729 >> 172.21.15.45:0/2572097729 conn(0x7f44b00ff300 msgr2=0x7f44b0101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 -- 172.21.15.45:0/2572097729 shutdown_connections 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.172+0000 7f44b4c34640 1 -- 172.21.15.45:0/2572097729 wait complete. 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44b4c34640 1 Processor -- start 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44b4c34640 1 -- start start 2024-09-21T01:54:29.212 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44b4c34640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 0x7f44b0199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44b4c34640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44b01091b0 0x7f44b0199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44b4c34640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7f44b0110900 con 0x7f44b0103870 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44b4c34640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7f44b0110780 con 0x7f44b01091b0 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 0x7f44b0199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44ae59c640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 0x7f44b0199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34596/0 (socket says 172.21.15.45:34596) 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.173+0000 7f44ae59c640 1 -- 172.21.15.45:0/4233425347 learned_addr learned my addr 172.21.15.45:0/4233425347 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44add9b640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44b01091b0 0x7f44b0199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44ae59c640 1 -- 172.21.15.45:0/4233425347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44b01091b0 msgr2=0x7f44b0199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44ae59c640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44b01091b0 0x7f44b0199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44ae59c640 1 -- 172.21.15.45:0/4233425347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b019f0d0 con 0x7f44b0103870 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44ae59c640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 0x7f44b0199890 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f44a000d6f0 tx=0x7f44a000dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f44a0017070 con 0x7f44b0103870 2024-09-21T01:54:29.213 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44b019f3c0 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7f44a000de00 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7f44a00049d0 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.174+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44b019f9a0 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.175+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44b0104af0 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.176+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7f44a0004cd0 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.176+0000 7f449f7fe640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f449807af30 0x7f449807d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.177+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7340+0+0 (secure 0 0 0) 0x7f44a009e270 con 0x7f44b0103870 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.177+0000 7f44add9b640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f449807af30 0x7f449807d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.177+0000 7f44add9b640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f449807af30 0x7f449807d440 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009230 tx=0x7f44a4004140 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:54:29.214 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.181+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7f44a00692a0 con 0x7f44b0103870 2024-09-21T01:54:29.384 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.383+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f44b0103c80 con 0x7f44b0103870 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.384+0000 7f449f7fe640 1 -- 172.21.15.45:0/4233425347 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v65) ==== 81+0+1009 (secure 0 0 0) 0x7f44a006e150 con 0x7f44b0103870 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6833/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1147989706 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1797066051 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3851449798 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:29.385 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3707884113 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3194963614 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6832/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1334988077 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1726148947 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/308996383 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1437053407 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:29.386 INFO:teuthology.orchestra.run.smithi045.stderr:listed 17 entries 2024-09-21T01:54:29.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f449807af30 msgr2=0x7f449807d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:29.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f449807af30 0x7f449807d440 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f44a4009230 tx=0x7f44a4004140 comp rx=0 tx=0).stop 2024-09-21T01:54:29.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 msgr2=0x7f44b0199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:29.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 0x7f44b0199890 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f44a000d6f0 tx=0x7f44a000dbb0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.387 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 shutdown_connections 2024-09-21T01:54:29.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7f449807af30 0x7f449807d440 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f44b01091b0 0x7f44b0199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 --2- 172.21.15.45:0/4233425347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f44b0103870 0x7f44b0199890 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:29.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 >> 172.21.15.45:0/4233425347 conn(0x7f44b00ff300 msgr2=0x7f44b0101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:54:29.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 shutdown_connections 2024-09-21T01:54:29.388 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:29.386+0000 7f44b4c34640 1 -- 172.21.15.45:0/4233425347 wait complete. 2024-09-21T01:54:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:29 smithi045 ceph-mon[28361]: pgmap v1001: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 189 KiB/s wr, 59 op/s 2024-09-21T01:54:30.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:29 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/4233425347' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:54:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:29 smithi090 ceph-mon[35142]: pgmap v1001: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 189 KiB/s wr, 59 op/s 2024-09-21T01:54:30.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:29 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/4233425347' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:54:30.345 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-21T01:54:30.345 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:54:30.366 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410 shell --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 -- ceph osd blocklist ls 2024-09-21T01:54:30.639 INFO:teuthology.orchestra.run.smithi045.stderr:Inferring config /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/mon.smithi045/config 2024-09-21T01:54:31.227 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 -- 172.21.15.45:0/3420604020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac102750 msgr2=0x7ff6ac102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 --2- 172.21.15.45:0/3420604020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac102750 0x7ff6ac102b60 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009880 tx=0x7ff69c02a4d0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 -- 172.21.15.45:0/3420604020 shutdown_connections 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 --2- 172.21.15.45:0/3420604020 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6ac108090 0x7ff6ac1084a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 --2- 172.21.15.45:0/3420604020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac102750 0x7ff6ac102b60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 -- 172.21.15.45:0/3420604020 >> 172.21.15.45:0/3420604020 conn(0x7ff6ac0fe1a0 msgr2=0x7ff6ac1005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 -- 172.21.15.45:0/3420604020 shutdown_connections 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 -- 172.21.15.45:0/3420604020 wait complete. 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 Processor -- start 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.198+0000 7ff6b3417640 1 -- start start 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b3417640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6ac102750 0x7ff6ac198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b3417640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 0x7ff6ac198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b3417640 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10f860 con 0x7ff6ac108090 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b3417640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac10f6e0 con 0x7ff6ac102750 2024-09-21T01:54:31.228 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 0x7ff6ac198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 0x7ff6ac198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:34622/0 (socket says 172.21.15.45:34622) 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 -- 172.21.15.45:0/691634341 learned_addr learned my addr 172.21.15.45:0/691634341 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b11b3640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6ac102750 0x7ff6ac198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 -- 172.21.15.45:0/691634341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6ac102750 msgr2=0x7ff6ac198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6ac102750 0x7ff6ac198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 -- 172.21.15.45:0/691634341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac06b990 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6b09b2640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 0x7ff6ac198ad0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0006fd0 tx=0x7ff6a000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.199+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff6a0015020 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.200+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac199010 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.200+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(25 keys) ==== 1063+0+0 (secure 0 0 0) 0x7ff6a0011ac0 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.200+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 ==== 357+0+0 (secure 0 0 0) 0x7ff6a0010b80 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.200+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac1a5c80 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.201+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac1039d0 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.201+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 19) ==== 103320+0+0 (secure 0 0 0) 0x7ff6a0018020 con 0x7ff6ac108090 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.202+0000 7ff6aa7fc640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff69407af30 0x7ff69407d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-21T01:54:31.229 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.202+0000 7ff6b11b3640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff69407af30 0x7ff69407d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-21T01:54:31.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.202+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 7340+0+0 (secure 0 0 0) 0x7ff6a009de90 con 0x7ff6ac108090 2024-09-21T01:54:31.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.203+0000 7ff6b11b3640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff69407af30 0x7ff69407d440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff69c000c00 tx=0x7ff69c005d50 comp rx=0 tx=0).ready entity=mgr.14227 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-21T01:54:31.230 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.207+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+198400 (secure 0 0 0) 0x7ff6a001f020 con 0x7ff6ac108090 2024-09-21T01:54:31.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.409+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff6ac10a830 con 0x7ff6ac108090 2024-09-21T01:54:31.411 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.410+0000 7ff6aa7fc640 1 -- 172.21.15.45:0/691634341 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v65) ==== 81+0+1009 (secure 0 0 0) 0x7ff6a0068f60 con 0x7ff6ac108090 2024-09-21T01:54:31.411 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6833/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1147989706 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1797066051 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3851449798 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3707884113 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/3194963614 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2275235431 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.90:6832/3876994677 2024-09-22T01:28:40.409775+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1334988077 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1726148947 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/308996383 2024-09-22T01:22:51.944078+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/3815412594 2024-09-22T01:21:11.091918+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6800/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:6801/2568383000 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:31.412 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45:0/1437053407 2024-09-22T01:21:45.172023+0000 2024-09-21T01:54:31.413 INFO:teuthology.orchestra.run.smithi045.stderr:listed 17 entries 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.412+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff69407af30 msgr2=0x7ff69407d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.412+0000 7ff6b3417640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff69407af30 0x7ff69407d440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff69c000c00 tx=0x7ff69c005d50 comp rx=0 tx=0).stop 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 msgr2=0x7ff6ac198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 0x7ff6ac198ad0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0006fd0 tx=0x7ff6a000b860 comp rx=0 tx=0).stop 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 shutdown_connections 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:6800/3847851980,v1:172.21.15.45:6801/3847851980] conn(0x7ff69407af30 0x7ff69407d440 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff6ac108090 0x7ff6ac198ad0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.414 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 --2- 172.21.15.45:0/691634341 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff6ac102750 0x7ff6ac198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-21T01:54:31.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 >> 172.21.15.45:0/691634341 conn(0x7ff6ac0fe1a0 msgr2=0x7ff6ac1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-21T01:54:31.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 shutdown_connections 2024-09-21T01:54:31.415 INFO:teuthology.orchestra.run.smithi045.stderr:2024-09-21T01:54:31.413+0000 7ff6b3417640 1 -- 172.21.15.45:0/691634341 wait complete. 2024-09-21T01:54:32.052 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:31 smithi045 ceph-mon[28361]: pgmap v1002: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 KiB/s wr, 34 op/s 2024-09-21T01:54:32.052 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:31 smithi045 ceph-mon[28361]: from='client.? 172.21.15.45:0/691634341' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:54:32.054 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-09-21T01:54:32.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:32.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-21T01:54:32.111 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T01:54:32.111 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi090.stderr:ceph-fuse[61750]: fuse finished with error 0 and tester_r 0 2024-09-21T01:54:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:31 smithi090 ceph-mon[35142]: pgmap v1002: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 KiB/s wr, 34 op/s 2024-09-21T01:54:32.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:31 smithi090 ceph-mon[35142]: from='client.? 172.21.15.45:0/691634341' entity='client.admin' cmd={"prefix": "osd blocklist ls"} : dispatch 2024-09-21T01:54:33.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:33 smithi090 ceph-mon[35142]: pgmap v1003: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-21T01:54:34.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:33 smithi045 ceph-mon[28361]: pgmap v1003: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 11 KiB/s wr, 16 op/s 2024-09-21T01:54:36.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:35 smithi045 ceph-mon[28361]: pgmap v1004: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 8.5 KiB/s wr, 2 op/s 2024-09-21T01:54:36.247 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:35 smithi090 ceph-mon[35142]: pgmap v1004: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 8.5 KiB/s wr, 2 op/s 2024-09-21T01:54:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:37 smithi045 ceph-mon[28361]: pgmap v1005: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 8.7 KiB/s wr, 2 op/s 2024-09-21T01:54:38.170 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:37 smithi045 ceph-mon[28361]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:38.213 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-09-21T01:54:38.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.213 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-21T01:54:38.248 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:37 smithi090 ceph-mon[35142]: pgmap v1005: 129 pgs: 129 active+clean; 228 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 255 B/s rd, 8.7 KiB/s wr, 2 op/s 2024-09-21T01:54:38.249 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:37 smithi090 ceph-mon[35142]: from='mgr.14227 172.21.15.45:0/962616127' entity='mgr.smithi045.txjudz' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T01:54:38.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.266 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-09-21T01:54:38.326 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-21T01:54:38.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-21T01:54:38.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-09-21T01:54:38.500 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-21T01:54:38.515 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-21T01:54:38.515 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-09-21T01:54:38.515 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-21T01:54:38.553 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056795,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=215 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/ad670c58f480d9e376cc1c6dba4c0607970d77491dc2bb482754055aa7bbd7c5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/ad670c58f480d9e376cc1c6dba4c0607970d77491dc2bb482754055aa7bbd7c5/diff,workdir=/var/lib/containers/storage/overlay/ad670c58f480d9e376cc1c6dba4c0607970d77491dc2bb482754055aa7bbd7c5/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/35d511838eb8060114365597eedc2ef8bad80d74dc65ca62d9657d5e2bf5357e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/35d511838eb8060114365597eedc2ef8bad80d74dc65ca62d9657d5e2bf5357e/diff,workdir=/var/lib/containers/storage/overlay/35d511838eb8060114365597eedc2ef8bad80d74dc65ca62d9657d5e2bf5357e/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249348k,nr_inodes=812337,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/b685952e2e2d878ba2ce6c441b8fd124b8454d248daafd031c38c59f3f31dda0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/b685952e2e2d878ba2ce6c441b8fd124b8454d248daafd031c38c59f3f31dda0/diff,workdir=/var/lib/containers/storage/overlay/b685952e2e2d878ba2ce6c441b8fd124b8454d248daafd031c38c59f3f31dda0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/99edfbb781b52f376b34046ccd185f933cb17ff2ab617733cce508cee86ac18a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/99edfbb781b52f376b34046ccd185f933cb17ff2ab617733cce508cee86ac18a/diff,workdir=/var/lib/containers/storage/overlay/99edfbb781b52f376b34046ccd185f933cb17ff2ab617733cce508cee86ac18a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5d145088b4ea1a0ecd81969e5445fbeae0e8294bdd2918360150e022e3c279ff/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/YCFD637RELDHJGJ4F5GKLUEQUQ:/var/lib/containers/storage/overlay/l/HOPQGXGSAQWI3ZP5PTHZNCXOZ7:/var/lib/containers/storage/overlay/l/NQHEEXUK6K5XKQD4KWIOOOBMPU,upperdir=/var/lib/containers/storage/overlay/5d145088b4ea1a0ecd81969e5445fbeae0e8294bdd2918360150e022e3c279ff/diff,workdir=/var/lib/containers/storage/overlay/5d145088b4ea1a0ecd81969e5445fbeae0e8294bdd2918360150e022e3c279ff/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/79723413cbceb56e2386ef1d7ea969a989b4ccd832c7c6e9ce3119bcb2b238e9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4VWD3FRWADQ5RWB4DQNXC375IT:/var/lib/containers/storage/overlay/l/VLSYNFJMRONSSH7FXSIY64I55R:/var/lib/containers/storage/overlay/l/NJNXDLWXDPHN5MJVM6YY6KPD7N:/var/lib/containers/storage/overlay/l/RH22B3JQUJMF4KWQYNRV6V5KQW:/var/lib/containers/storage/overlay/l/IGFCDNJGACSCBFD36VXNMQYJ4Q:/var/lib/containers/storage/overlay/l/G4JWT5MPSBBHOJB7IFMQWXT7CE:/var/lib/containers/storage/overlay/l/CWL76UYEBWYSJZOSZYB7DII327,upperdir=/var/lib/containers/storage/overlay/79723413cbceb56e2386ef1d7ea969a989b4ccd832c7c6e9ce3119bcb2b238e9/diff,workdir=/var/lib/containers/storage/overlay/79723413cbceb56e2386ef1d7ea969a989b4ccd832c7c6e9ce3119bcb2b238e9/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d0ffbccd542a013b862dad4964c062f1a5191cda9957ad8bfba3f0368458ef9d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/H2GJ2F3ZEBL5VWHBCQSIU4QB45:/var/lib/containers/storage/overlay/l/VN2NUDAZGL5ZWO5VQG7ORJB5GD:/var/lib/containers/storage/overlay/l/7UT7ZIUJBSTJUNGUVZ3KUMM4PJ:/var/lib/containers/storage/overlay/l/DE2P6R4GZNWRHQTXNWAY7EMQQI:/var/lib/containers/storage/overlay/l/5FYLQTKF35KS36T2272PACB6II:/var/lib/containers/storage/overlay/l/HUWBEI3WLOEPXD5Q2F5DKHIAIO:/var/lib/containers/storage/overlay/l/B2RPMVD35PIU6L7PLPBGFQHXGC:/var/lib/containers/storage/overlay/l/IMGU3ILO733RQK4KS6XWWDM3XV:/var/lib/containers/storage/overlay/l/IYLJXEUF5ISSRXOKYZW4YSFLDK:/var/lib/containers/storage/overlay/l/Q6PORZRAM4QT254LNI23N2RJBF,upperdir=/var/lib/containers/storage/overlay/d0ffbccd542a013b862dad4964c062f1a5191cda9957ad8bfba3f0368458ef9d/diff,workdir=/var/lib/containers/storage/overlay/d0ffbccd542a013b862dad4964c062f1a5191cda9957ad8bfba3f0368458ef9d/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9c05e27e6636bda2c12edddeda46732180428a3e53b4f572c0f998f19a0061fe/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4IBOBKGJDP5PRHZZ2E3GO3RGJZ:/var/lib/containers/storage/overlay/l/OKPYZH6ONAKS7F23U2B6JS3DZD:/var/lib/containers/storage/overlay/l/3T5DBHEBY5RZBAHNVFJWUM4KG2:/var/lib/containers/storage/overlay/l/3GHDGHLPOE2XG2LPEPJRU3BNDQ:/var/lib/containers/storage/overlay/l/G6PEVTZ5FRS3GICXJTQ5QSTIZ2:/var/lib/containers/storage/overlay/l/Q3P4EDTBWMPZVALJM7746VGTZH:/var/lib/containers/storage/overlay/l/QND4RLQPYILRP7GOVQD6CU54Z4:/var/lib/containers/storage/overlay/l/HEISMK5NBZ56PGCHSF77LCDVVS:/var/lib/containers/storage/overlay/l/ADCMLB2YAL2IPODFA6UKFBSG4H:/var/lib/containers/storage/overlay/l/73XLG4SAHJZDFDCUX6SKSGGVQW:/var/lib/containers/storage/overlay/l/G4JWT5MPSBBHOJB7IFMQWXT7CE:/var/lib/containers/storage/overlay/l/CWL76UYEBWYSJZOSZYB7DII327,upperdir=/var/lib/containers/storage/overlay/9c05e27e6636bda2c12edddeda46732180428a3e53b4f572c0f998f19a0061fe/diff,workdir=/var/lib/containers/storage/overlay/9c05e27e6636bda2c12edddeda46732180428a3e53b4f572c0f998f19a0061fe/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a1b6cdebe689e71c429418fe5326524897937b5f37d90de992db69d64fa44060/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/a1b6cdebe689e71c429418fe5326524897937b5f37d90de992db69d64fa44060/diff,workdir=/var/lib/containers/storage/overlay/a1b6cdebe689e71c429418fe5326524897937b5f37d90de992db69d64fa44060/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5e89a7de0fbad5d96bbf74acf5b444bb3e989b0f6bac88654a894b9213faee69/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/5e89a7de0fbad5d96bbf74acf5b444bb3e989b0f6bac88654a894b9213faee69/diff,workdir=/var/lib/containers/storage/overlay/5e89a7de0fbad5d96bbf74acf5b444bb3e989b0f6bac88654a894b9213faee69/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/07e5a5b141b806d2d20c0b579de4ac1744b52f19d9d69ade702121e314cfae92/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/07e5a5b141b806d2d20c0b579de4ac1744b52f19d9d69ade702121e314cfae92/diff,workdir=/var/lib/containers/storage/overlay/07e5a5b141b806d2d20c0b579de4ac1744b52f19d9d69ade702121e314cfae92/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/975651874b44ae6cb409a8d281f45f482bf50affe77b4a3d2a0a47dea2d33c73/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/975651874b44ae6cb409a8d281f45f482bf50affe77b4a3d2a0a47dea2d33c73/diff,workdir=/var/lib/containers/storage/overlay/975651874b44ae6cb409a8d281f45f482bf50affe77b4a3d2a0a47dea2d33c73/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/65f1d527baeba808c191f87fbbfe6f352d1b3242aa722cbf123a3082635ab3cc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/65f1d527baeba808c191f87fbbfe6f352d1b3242aa722cbf123a3082635ab3cc/diff,workdir=/var/lib/containers/storage/overlay/65f1d527baeba808c191f87fbbfe6f352d1b3242aa722cbf123a3082635ab3cc/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/7f9468bc312dfe95d70fe38c47c2bf1b307a698f5a390af2e5ef5bbb0aa68985/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z5KAPADVKIFXUYWC4DA734KUM4:/var/lib/containers/storage/overlay/l/RUDITZXLWWKTTUYWNOCFRI6CZ2,upperdir=/var/lib/containers/storage/overlay/7f9468bc312dfe95d70fe38c47c2bf1b307a698f5a390af2e5ef5bbb0aa68985/diff,workdir=/var/lib/containers/storage/overlay/7f9468bc312dfe95d70fe38c47c2bf1b307a698f5a390af2e5ef5bbb0aa68985/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-09-21T01:54:38.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.554 DEBUG:teuthology.orchestra.run.smithi045:> ip netns list 2024-09-21T01:54:38.614 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-21T01:54:38.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.614 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-21T01:54:38.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-21T01:54:38.689 DEBUG:teuthology.orchestra.run.smithi045:> sudo ip link delete ceph-brx 2024-09-21T01:54:38.795 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T01:54:38.804 INFO:tasks.cephadm:Teardown begin 2024-09-21T01:54:38.805 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:54:38.847 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:54:38.879 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T01:54:38.879 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:54:38.904 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T01:54:38.935 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T01:54:38.935 INFO:tasks.cephadm.mon.smithi045:Stopping mon.smithi045... 2024-09-21T01:54:38.936 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045 2024-09-21T01:54:39.254 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:38 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:54:39.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:39 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045[28357]: 2024-09-21T01:54:39.253+0000 7fa5f1e57640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:54:39.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:39 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045[28357]: 2024-09-21T01:54:39.253+0000 7fa5f1e57640 -1 mon.smithi045@0(leader) e2 *** Got Signal Terminated *** 2024-09-21T01:54:39.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:39 smithi045 podman[85587]: 2024-09-21 01:54:39.33568283 +0000 UTC m=+0.186864235 container died 387dd03e82eb0c7ee167de2f8f55e31436428e4b9d384b3f6f69a5560267fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:54:39.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:39 smithi045 podman[85587]: 2024-09-21 01:54:39.745959442 +0000 UTC m=+0.597140846 container cleanup 387dd03e82eb0c7ee167de2f8f55e31436428e4b9d384b3f6f69a5560267fb40 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1843221895adccfb3b692060251dd6a0183bc410, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T01:54:39.920 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Sep 21 01:54:39 smithi045 bash[85587]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi045 2024-09-21T01:54:40.109 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi045.service' 2024-09-21T01:54:40.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:54:40.167 INFO:tasks.cephadm.mon.smithi045:Stopped mon.smithi045 2024-09-21T01:54:40.167 INFO:tasks.cephadm.mon.smithi090:Stopping mon.smithi090... 2024-09-21T01:54:40.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi090 2024-09-21T01:54:40.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:40 smithi090 systemd[1]: Stopping Ceph mon.smithi090 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:54:40.998 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090[35117]: 2024-09-21T01:54:40.572+0000 7f3dd307e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi090 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:54:40.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090[35117]: 2024-09-21T01:54:40.572+0000 7f3dd307e640 -1 mon.smithi090@1(peon) e2 *** Got Signal Terminated *** 2024-09-21T01:54:40.999 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:40 smithi090 podman[67283]: 2024-09-21 01:54:40.767603545 +0000 UTC m=+0.309080649 container died a7fe49195229eab87bec6279d5b7d101ac2a669ffe3fa9e3dde08fe8e3dd9fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T01:54:41.498 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:41 smithi090 podman[67283]: 2024-09-21 01:54:41.275037447 +0000 UTC m=+0.816514551 container cleanup a7fe49195229eab87bec6279d5b7d101ac2a669ffe3fa9e3dde08fe8e3dd9fd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T01:54:41.499 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 21 01:54:41 smithi090 bash[67283]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-mon-smithi090 2024-09-21T01:54:41.682 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@mon.smithi090.service' 2024-09-21T01:54:41.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:54:41.734 INFO:tasks.cephadm.mon.smithi090:Stopped mon.smithi090 2024-09-21T01:54:41.734 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T01:54:41.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.0 2024-09-21T01:54:42.026 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:41 smithi045 systemd[1]: Stopping Ceph osd.0 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:54:42.420 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:42 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0[48701]: 2024-09-21T01:54:42.025+0000 7f8630882640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:54:42.420 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:42 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0[48701]: 2024-09-21T01:54:42.025+0000 7f8630882640 -1 osd.0 65 *** Got signal Terminated *** 2024-09-21T01:54:42.420 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:42 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0[48701]: 2024-09-21T01:54:42.025+0000 7f8630882640 -1 osd.0 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:54:47.812 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:47 smithi045 podman[85720]: 2024-09-21 01:54:47.503946276 +0000 UTC m=+5.572380379 container died dbba428d3abad9244379910304b029c91aadbf07ba2a54bf428fb796dd0c2ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T01:54:48.386 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 podman[85720]: 2024-09-21 01:54:48.134933196 +0000 UTC m=+6.203367245 container cleanup dbba428d3abad9244379910304b029c91aadbf07ba2a54bf428fb796dd0c2ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T01:54:48.386 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 bash[85720]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0 2024-09-21T01:54:48.669 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 podman[85737]: 2024-09-21 01:54:48.386527353 +0000 UTC m=+0.852032781 container remove dbba428d3abad9244379910304b029c91aadbf07ba2a54bf428fb796dd0c2ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T01:54:49.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 podman[85826]: 2024-09-21 01:54:48.608238026 +0000 UTC m=+0.016507981 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:54:49.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 podman[85826]: 2024-09-21 01:54:48.71268023 +0000 UTC m=+0.120950189 container create 10c01facc259adc130b5327000e7d9ebc12a9702d41a9196938cb4c5384a4230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T01:54:49.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 podman[85826]: 2024-09-21 01:54:48.923997155 +0000 UTC m=+0.332267116 container init 10c01facc259adc130b5327000e7d9ebc12a9702d41a9196938cb4c5384a4230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T01:54:49.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:48 smithi045 podman[85826]: 2024-09-21 01:54:48.927493354 +0000 UTC m=+0.335763318 container start 10c01facc259adc130b5327000e7d9ebc12a9702d41a9196938cb4c5384a4230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T01:54:49.007 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:49 smithi045 podman[85826]: 2024-09-21 01:54:49.00736863 +0000 UTC m=+0.415638594 container attach 10c01facc259adc130b5327000e7d9ebc12a9702d41a9196938cb4c5384a4230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T01:54:49.670 INFO:journalctl@ceph.osd.0.smithi045.stdout:Sep 21 01:54:49 smithi045 podman[85826]: 2024-09-21 01:54:49.304290894 +0000 UTC m=+0.712560855 container died 10c01facc259adc130b5327000e7d9ebc12a9702d41a9196938cb4c5384a4230 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T01:54:49.910 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.0.service' 2024-09-21T01:54:49.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:54:49.951 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T01:54:49.951 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T01:54:49.951 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.1 2024-09-21T01:54:50.291 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:50 smithi045 systemd[1]: Stopping Ceph osd.1 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:54:50.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:50 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1[55451]: 2024-09-21T01:54:50.289+0000 7ffad0c7c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:54:50.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:50 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1[55451]: 2024-09-21T01:54:50.289+0000 7ffad0c7c640 -1 osd.1 65 *** Got signal Terminated *** 2024-09-21T01:54:50.670 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:50 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1[55451]: 2024-09-21T01:54:50.289+0000 7ffad0c7c640 -1 osd.1 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:54:55.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:55 smithi045 podman[85929]: 2024-09-21 01:54:55.654650527 +0000 UTC m=+5.472049853 container died 5fa2b9b5a6ce653b6adfd129a086ff1780dba8aa421f7ec405534e0b880c629e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T01:54:56.291 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[85929]: 2024-09-21 01:54:56.056839829 +0000 UTC m=+5.874239199 container cleanup 5fa2b9b5a6ce653b6adfd129a086ff1780dba8aa421f7ec405534e0b880c629e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:54:56.291 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 bash[85929]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1 2024-09-21T01:54:56.634 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[85944]: 2024-09-21 01:54:56.291509265 +0000 UTC m=+0.632246641 container remove 5fa2b9b5a6ce653b6adfd129a086ff1780dba8aa421f7ec405534e0b880c629e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T01:54:56.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[86038]: 2024-09-21 01:54:56.634175345 +0000 UTC m=+0.108578578 container create 32bad85bcba1fc46663462c710a87ef7ea5bd06c746a50f06481b446ef94f63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T01:54:56.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[86038]: 2024-09-21 01:54:56.542665853 +0000 UTC m=+0.017069097 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:54:56.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[86038]: 2024-09-21 01:54:56.845093897 +0000 UTC m=+0.319497141 container init 32bad85bcba1fc46663462c710a87ef7ea5bd06c746a50f06481b446ef94f63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_BRANCH=HEAD) 2024-09-21T01:54:56.920 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[86038]: 2024-09-21 01:54:56.848459568 +0000 UTC m=+0.322862801 container start 32bad85bcba1fc46663462c710a87ef7ea5bd06c746a50f06481b446ef94f63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:54:57.341 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:56 smithi045 podman[86038]: 2024-09-21 01:54:56.944036242 +0000 UTC m=+0.418439507 container attach 32bad85bcba1fc46663462c710a87ef7ea5bd06c746a50f06481b446ef94f63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:54:57.341 INFO:journalctl@ceph.osd.1.smithi045.stdout:Sep 21 01:54:57 smithi045 podman[86038]: 2024-09-21 01:54:57.038822128 +0000 UTC m=+0.513225372 container died 32bad85bcba1fc46663462c710a87ef7ea5bd06c746a50f06481b446ef94f63f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:54:57.653 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.1.service' 2024-09-21T01:54:57.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:54:57.698 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T01:54:57.698 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T01:54:57.698 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.2 2024-09-21T01:54:57.919 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:54:57 smithi045 systemd[1]: Stopping Ceph osd.2 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:54:58.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:54:58 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2[61746]: 2024-09-21T01:54:58.020+0000 7fbb8baa5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:54:58.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:54:58 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2[61746]: 2024-09-21T01:54:58.020+0000 7fbb8baa5640 -1 osd.2 65 *** Got signal Terminated *** 2024-09-21T01:54:58.420 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:54:58 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2[61746]: 2024-09-21T01:54:58.020+0000 7fbb8baa5640 -1 osd.2 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:55:03.670 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:03 smithi045 podman[86140]: 2024-09-21 01:55:03.362820326 +0000 UTC m=+5.475826466 container died a6a30c7bac12c99aa082e745973b986beab7cd4e221182fb52a97677168c5c29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T01:55:04.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:03 smithi045 podman[86140]: 2024-09-21 01:55:03.795511211 +0000 UTC m=+5.908517341 container cleanup a6a30c7bac12c99aa082e745973b986beab7cd4e221182fb52a97677168c5c29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-21T01:55:04.064 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:03 smithi045 bash[86140]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2 2024-09-21T01:55:04.381 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86169]: 2024-09-21 01:55:04.064025042 +0000 UTC m=+0.699524325 container remove a6a30c7bac12c99aa082e745973b986beab7cd4e221182fb52a97677168c5c29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2) 2024-09-21T01:55:04.638 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86256]: 2024-09-21 01:55:04.382277542 +0000 UTC m=+0.016897637 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:55:04.638 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86256]: 2024-09-21 01:55:04.498429052 +0000 UTC m=+0.133049135 container create 8bbfc6e384a0f9db4d83bf7d673d3f96a95733ab83df472ae13e2eb4dafbe4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2-deactivate, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:55:04.920 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86256]: 2024-09-21 01:55:04.767996766 +0000 UTC m=+0.402616859 container init 8bbfc6e384a0f9db4d83bf7d673d3f96a95733ab83df472ae13e2eb4dafbe4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T01:55:04.920 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86256]: 2024-09-21 01:55:04.772815026 +0000 UTC m=+0.407435110 container start 8bbfc6e384a0f9db4d83bf7d673d3f96a95733ab83df472ae13e2eb4dafbe4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True) 2024-09-21T01:55:04.920 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86256]: 2024-09-21 01:55:04.842873528 +0000 UTC m=+0.477493626 container attach 8bbfc6e384a0f9db4d83bf7d673d3f96a95733ab83df472ae13e2eb4dafbe4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:55:05.263 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:04 smithi045 podman[86256]: 2024-09-21 01:55:04.978534178 +0000 UTC m=+0.613154283 container died 8bbfc6e384a0f9db4d83bf7d673d3f96a95733ab83df472ae13e2eb4dafbe4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T01:55:05.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:05 smithi045 podman[86280]: 2024-09-21 01:55:05.496822776 +0000 UTC m=+0.511338984 container remove 8bbfc6e384a0f9db4d83bf7d673d3f96a95733ab83df472ae13e2eb4dafbe4a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:55:05.513 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:05 smithi045 systemd[1]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-21T01:55:05.514 INFO:journalctl@ceph.osd.2.smithi045.stdout:Sep 21 01:55:05 smithi045 systemd[1]: Stopped Ceph osd.2 for 9b2fd594-77b7-11ef-baf5-efdc52797490. 2024-09-21T01:55:05.517 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.2.service' 2024-09-21T01:55:05.561 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:55:05.561 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T01:55:05.561 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T01:55:05.562 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.3 2024-09-21T01:55:05.920 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:05 smithi045 systemd[1]: Stopping Ceph osd.3 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:55:06.420 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:05 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:55:05.966+0000 7ff0a4e9a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:55:06.420 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:05 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:55:05.966+0000 7ff0a4e9a640 -1 osd.3 65 *** Got signal Terminated *** 2024-09-21T01:55:06.420 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:05 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:55:05.966+0000 7ff0a4e9a640 -1 osd.3 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:55:09.920 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:09 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:55:09.601+0000 7ff0a1485640 -1 osd.3 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.126641+0000 front 2024-09-21T01:54:44.126767+0000 (oldest deadline 2024-09-21T01:55:08.826788+0000) 2024-09-21T01:55:10.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:09 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:09.783+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:10.920 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:10 smithi045 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3[68167]: 2024-09-21T01:55:10.604+0000 7ff0a1485640 -1 osd.3 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.126641+0000 front 2024-09-21T01:54:44.126767+0000 (oldest deadline 2024-09-21T01:55:08.826788+0000) 2024-09-21T01:55:11.249 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:10 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:10.800+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:11.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:10 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:10.775+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:11.743 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:11 smithi045 podman[86359]: 2024-09-21 01:55:11.490196102 +0000 UTC m=+5.689981457 container died 42791864b1599f43c720395cda2d1d6c4960d158b1f2e3dc11b5811069973fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:55:12.012 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:11 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:11.756+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:12.012 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:11 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:11.816+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:12.143 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:11 smithi045 podman[86359]: 2024-09-21 01:55:11.841633238 +0000 UTC m=+6.041418587 container cleanup 42791864b1599f43c720395cda2d1d6c4960d158b1f2e3dc11b5811069973fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:55:12.143 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:11 smithi045 bash[86359]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3 2024-09-21T01:55:12.420 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86377]: 2024-09-21 01:55:12.142983693 +0000 UTC m=+0.650245059 container remove 42791864b1599f43c720395cda2d1d6c4960d158b1f2e3dc11b5811069973fcb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T01:55:12.746 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86463]: 2024-09-21 01:55:12.365436959 +0000 UTC m=+0.016461023 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:55:12.747 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86463]: 2024-09-21 01:55:12.485761002 +0000 UTC m=+0.136785094 container create 1d38cc02b750066760ab44b1272723c52744e42c5b87c8692d082e20b52d930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T01:55:12.747 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86463]: 2024-09-21 01:55:12.746940589 +0000 UTC m=+0.397964650 container init 1d38cc02b750066760ab44b1272723c52744e42c5b87c8692d082e20b52d930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T01:55:12.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:12 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:12.418+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:13.170 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86463]: 2024-09-21 01:55:12.750425054 +0000 UTC m=+0.401449105 container start 1d38cc02b750066760ab44b1272723c52744e42c5b87c8692d082e20b52d930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:55:13.170 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86463]: 2024-09-21 01:55:12.846994081 +0000 UTC m=+0.498018133 container attach 1d38cc02b750066760ab44b1272723c52744e42c5b87c8692d082e20b52d930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T01:55:13.170 INFO:journalctl@ceph.osd.3.smithi045.stdout:Sep 21 01:55:12 smithi045 podman[86463]: 2024-09-21 01:55:12.949040569 +0000 UTC m=+0.600064629 container died 1d38cc02b750066760ab44b1272723c52744e42c5b87c8692d082e20b52d930c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:55:13.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:12 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:12.799+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:13.248 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:12 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:12.857+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:13.594 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.3.service' 2024-09-21T01:55:13.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:55:13.639 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T01:55:13.639 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T01:55:13.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.4 2024-09-21T01:55:13.669 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:13.380+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:13.960 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:13.875+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:13.960 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:13.809+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:13.960 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:13 smithi090 systemd[1]: Stopping Ceph osd.4 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:55:13.960 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:13.773+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:14.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:13.959+0000 7f7c75e29640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:55:14.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:13.959+0000 7f7c75e29640 -1 osd.4 65 *** Got signal Terminated *** 2024-09-21T01:55:14.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:13 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:13.959+0000 7f7c75e29640 -1 osd.4 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:55:14.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:14 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:14.348+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:15.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:14 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:14.790+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:15.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:14 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:14.827+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:15.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:14 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:14.775+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:15.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:15 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:15.314+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:16.249 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:15 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:15.791+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:16.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:15 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:15.852+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:16.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:15 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:15.773+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:16.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:16 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:16.337+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:17.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:16 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:16.793+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:17.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:16 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:16.853+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:17.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:16 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:16.776+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:17.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:16 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:16.776+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:17.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:17.348+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:17.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:17.348+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:18.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:17.784+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:18.249 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:17.784+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:55.244365+0000 front 2024-09-21T01:54:55.244532+0000 (oldest deadline 2024-09-21T01:55:17.544451+0000) 2024-09-21T01:55:18.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:17.888+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:18.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:17.787+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:18.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:17 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:17.787+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:18.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:18.358+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:18.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:18.358+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:19.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:18.790+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:44.743159+0000 front 2024-09-21T01:54:44.743171+0000 (oldest deadline 2024-09-21T01:55:10.643023+0000) 2024-09-21T01:55:19.249 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4[39870]: 2024-09-21T01:55:18.790+0000 7f7c72414640 -1 osd.4 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:55.244365+0000 front 2024-09-21T01:54:55.244532+0000 (oldest deadline 2024-09-21T01:55:17.544451+0000) 2024-09-21T01:55:19.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:18.904+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:19.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:18.904+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:19.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:18.757+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:19.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:18 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:18.757+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:19.619 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:19 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:19.331+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:19.620 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:19 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:19.331+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:19.620 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:19 smithi090 podman[67419]: 2024-09-21 01:55:19.3500451 +0000 UTC m=+5.505119251 container died 281d55ea95e9c55877c57e48fd09a0da8bb0b74d74141d621db2182e9132dd60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T01:55:19.961 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:19 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:19.865+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:19.961 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:19 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:19.865+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:19.962 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:19 smithi090 podman[67419]: 2024-09-21 01:55:19.709944267 +0000 UTC m=+5.865018399 container cleanup 281d55ea95e9c55877c57e48fd09a0da8bb0b74d74141d621db2182e9132dd60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-21T01:55:19.962 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:19 smithi090 bash[67419]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4 2024-09-21T01:55:19.962 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:19 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:19.764+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:19.962 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:19 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:19.764+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:20.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:19 smithi090 podman[67436]: 2024-09-21 01:55:19.961417014 +0000 UTC m=+0.606810671 container remove 281d55ea95e9c55877c57e48fd09a0da8bb0b74d74141d621db2182e9132dd60 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-21T01:55:20.575 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:20 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:20.354+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:20.575 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:20 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:20.354+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:20.575 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:20 smithi090 podman[67524]: 2024-09-21 01:55:20.180254985 +0000 UTC m=+0.015801219 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:55:20.576 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:20 smithi090 podman[67524]: 2024-09-21 01:55:20.321045638 +0000 UTC m=+0.156591842 container create 65570f8e39e70e9ec9e90b72d31b03db5ef75401f03075234704bafb66458a25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux ) 2024-09-21T01:55:20.576 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:20 smithi090 podman[67524]: 2024-09-21 01:55:20.575339729 +0000 UTC m=+0.410885942 container init 65570f8e39e70e9ec9e90b72d31b03db5ef75401f03075234704bafb66458a25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T01:55:20.979 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:20 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:20.821+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:20.980 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:20 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:20.821+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:20.980 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:20 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:20.809+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:20.980 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:20 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:20.809+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:20.980 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:20 smithi090 podman[67524]: 2024-09-21 01:55:20.5794399 +0000 UTC m=+0.414986104 container start 65570f8e39e70e9ec9e90b72d31b03db5ef75401f03075234704bafb66458a25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T01:55:20.980 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:20 smithi090 podman[67524]: 2024-09-21 01:55:20.69267609 +0000 UTC m=+0.528222403 container attach 65570f8e39e70e9ec9e90b72d31b03db5ef75401f03075234704bafb66458a25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T01:55:21.248 INFO:journalctl@ceph.osd.4.smithi090.stdout:Sep 21 01:55:20 smithi090 podman[67524]: 2024-09-21 01:55:20.98008252 +0000 UTC m=+0.815628745 container died 65570f8e39e70e9ec9e90b72d31b03db5ef75401f03075234704bafb66458a25 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T01:55:21.505 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:21.364+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:21.506 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:21.364+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:21.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.4.service' 2024-09-21T01:55:21.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:55:21.609 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T01:55:21.609 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T01:55:21.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.5 2024-09-21T01:55:21.787 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:21 smithi090 systemd[1]: Stopping Ceph osd.5 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:21.810+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:21.810+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:21.852+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:21.852+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:21.942+0000 7efddf380640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:21.942+0000 7efddf380640 -1 osd.5 65 *** Got signal Terminated *** 2024-09-21T01:55:22.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:21 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:21.942+0000 7efddf380640 -1 osd.5 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:55:22.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:22 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:22.341+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:22.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:22 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:22.341+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:23.248 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:22 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:22.805+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:23.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:22 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:22.805+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:23.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:22 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:22.782+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:23.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:22 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:22.782+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:23.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:23.335+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:23.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:23.335+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:24.248 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:23.793+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:24.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:23.793+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:24.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:23.793+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:24.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:23.852+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:24.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:23 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:23.852+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:24.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:24.378+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:24.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:24.378+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:25.248 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:24.756+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:25.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:24.756+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:25.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:24.756+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:25.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:24.814+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:25.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:24.814+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:25.249 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:24 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:24.814+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:55:02.753963+0000 front 2024-09-21T01:55:02.754031+0000 (oldest deadline 2024-09-21T01:55:24.453962+0000) 2024-09-21T01:55:25.733 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:25.417+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:25.733 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:25.417+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:25.733 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:25.417+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:25.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:25.732+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:25.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:25.732+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:25.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:25.732+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:25.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:25.857+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:25.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:25.857+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:25.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:25 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:25.857+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:55:02.753963+0000 front 2024-09-21T01:55:02.754031+0000 (oldest deadline 2024-09-21T01:55:24.453962+0000) 2024-09-21T01:55:26.689 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:26.466+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:26.689 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:26.466+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:26.689 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:26.466+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:26.998 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:26.688+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:26.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:26.688+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:26.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:26.688+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:26.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:26.815+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:45.251531+0000 front 2024-09-21T01:54:45.251738+0000 (oldest deadline 2024-09-21T01:55:09.351641+0000) 2024-09-21T01:55:26.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:26.815+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:52.753271+0000 front 2024-09-21T01:54:52.753221+0000 (oldest deadline 2024-09-21T01:55:18.653151+0000) 2024-09-21T01:55:26.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:26 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5[45289]: 2024-09-21T01:55:26.815+0000 7efddb96b640 -1 osd.5 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:55:02.753963+0000 front 2024-09-21T01:55:02.754031+0000 (oldest deadline 2024-09-21T01:55:24.453962+0000) 2024-09-21T01:55:27.745 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:27 smithi090 podman[67628]: 2024-09-21 01:55:27.499145167 +0000 UTC m=+5.656501460 container died 658b7398123a30d9fa2a26b75f702d9e29a13f8b622a21bcf993c0abd986f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-21T01:55:27.746 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:27 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:27.711+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:27.746 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:27 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:27.711+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:27.746 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:27 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:27.711+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:27.746 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:27 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:27.468+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:27.746 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:27 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:27.468+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:27.746 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:27 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:27.468+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:27.998 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:27 smithi090 podman[67628]: 2024-09-21 01:55:27.852880857 +0000 UTC m=+6.010237145 container cleanup 658b7398123a30d9fa2a26b75f702d9e29a13f8b622a21bcf993c0abd986f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T01:55:27.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:27 smithi090 bash[67628]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5 2024-09-21T01:55:28.435 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67653]: 2024-09-21 01:55:28.1294102 +0000 UTC m=+0.623356803 container remove 658b7398123a30d9fa2a26b75f702d9e29a13f8b622a21bcf993c0abd986f80a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2) 2024-09-21T01:55:28.686 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67741]: 2024-09-21 01:55:28.455556221 +0000 UTC m=+0.105221709 container create f638242d3a3bd9f9f355a3a2ec9f71307885a6b92c4ea04b8de47b724558aeeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-21T01:55:28.686 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67741]: 2024-09-21 01:55:28.36595242 +0000 UTC m=+0.015617924 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:55:28.686 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67741]: 2024-09-21 01:55:28.686418818 +0000 UTC m=+0.336084311 container init f638242d3a3bd9f9f355a3a2ec9f71307885a6b92c4ea04b8de47b724558aeeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:55:28.687 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:28 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:28.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:28.687 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:28 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:28.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:28.687 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:28 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:28.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:28.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67741]: 2024-09-21 01:55:28.689735021 +0000 UTC m=+0.339400510 container start f638242d3a3bd9f9f355a3a2ec9f71307885a6b92c4ea04b8de47b724558aeeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T01:55:28.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67741]: 2024-09-21 01:55:28.796071486 +0000 UTC m=+0.445736981 container attach f638242d3a3bd9f9f355a3a2ec9f71307885a6b92c4ea04b8de47b724558aeeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:55:28.999 INFO:journalctl@ceph.osd.5.smithi090.stdout:Sep 21 01:55:28 smithi090 podman[67741]: 2024-09-21 01:55:28.883226887 +0000 UTC m=+0.532892376 container died f638242d3a3bd9f9f355a3a2ec9f71307885a6b92c4ea04b8de47b724558aeeb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T01:55:28.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:28 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:28.713+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:28.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:28 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:28.713+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:28.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:28 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:28.713+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:29.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:29.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:29.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:29.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:29.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:29.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:29.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.5.service' 2024-09-21T01:55:29.562 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:55:29.563 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T01:55:29.563 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T01:55:29.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.6 2024-09-21T01:55:29.762 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 systemd[1]: Stopping Ceph osd.6 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:55:30.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:29.760+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:30.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:29.760+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:30.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:29.760+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:30.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:29.869+0000 7fb27d674640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:55:30.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:29.869+0000 7fb27d674640 -1 osd.6 65 *** Got signal Terminated *** 2024-09-21T01:55:30.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:29 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:29.869+0000 7fb27d674640 -1 osd.6 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:55:30.731 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:30 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:30.383+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:30.731 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:30 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:30.383+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:30.731 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:30 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:30.383+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:30.998 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:30 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:30.730+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:30.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:30 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:30.730+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:30.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:30 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:30.730+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:31.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:31 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:31.372+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:31.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:31 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:31.372+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:31.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:31 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:31.372+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:32.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:31 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:31.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:32.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:31 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:31.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:32.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:31 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:31.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:32.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:32 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:32.343+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:32.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:32 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:32.343+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:32.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:32 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:32.343+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:33.248 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:32 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:32.764+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:33.248 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:32 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:32.764+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:33.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:32 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:32.764+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:33.754 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:33.392+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:33.754 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:33.392+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:33.754 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:33.392+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:33.754 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:33.392+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:34.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:33.810+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:34.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:33.810+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:34.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:33.810+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:34.249 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:33 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:33.810+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:07.556594+0000 front 2024-09-21T01:55:07.556580+0000 (oldest deadline 2024-09-21T01:55:32.856352+0000) 2024-09-21T01:55:34.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:34.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:34.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:34.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:34.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:34.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:34.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:34.396+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:35.229 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:34.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.652876+0000 front 2024-09-21T01:54:46.652954+0000 (oldest deadline 2024-09-21T01:55:13.053418+0000) 2024-09-21T01:55:35.229 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:34.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:53.054032+0000 front 2024-09-21T01:54:53.054202+0000 (oldest deadline 2024-09-21T01:55:15.954042+0000) 2024-09-21T01:55:35.229 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:34.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.455009+0000 front 2024-09-21T01:54:59.454973+0000 (oldest deadline 2024-09-21T01:55:23.554938+0000) 2024-09-21T01:55:35.229 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:34 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6[50797]: 2024-09-21T01:55:34.763+0000 7fb275c57640 -1 osd.6 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:07.556594+0000 front 2024-09-21T01:55:07.556580+0000 (oldest deadline 2024-09-21T01:55:32.856352+0000) 2024-09-21T01:55:35.499 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:35 smithi090 podman[67846]: 2024-09-21 01:55:35.229043409 +0000 UTC m=+5.439074816 container died d64824ee31d6ea9ef2b3b53f621b6cceb926aef1bdebcf89bece699d71278579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221) 2024-09-21T01:55:35.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:35 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:35.431+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:35.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:35 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:35.431+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:35.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:35 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:35.431+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:35.499 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:35 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:35.431+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:35.856 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:35 smithi090 podman[67846]: 2024-09-21 01:55:35.631089146 +0000 UTC m=+5.841120541 container cleanup d64824ee31d6ea9ef2b3b53f621b6cceb926aef1bdebcf89bece699d71278579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T01:55:35.857 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:35 smithi090 bash[67846]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6 2024-09-21T01:55:36.248 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:35 smithi090 podman[67861]: 2024-09-21 01:55:35.856266496 +0000 UTC m=+0.628490994 container remove d64824ee31d6ea9ef2b3b53f621b6cceb926aef1bdebcf89bece699d71278579 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:36 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:36.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:36 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:36.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:36 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:36.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:36 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:36.434+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:36 smithi090 podman[67950]: 2024-09-21 01:55:36.205967127 +0000 UTC m=+0.015937776 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:36 smithi090 podman[67950]: 2024-09-21 01:55:36.307489816 +0000 UTC m=+0.117460456 container create ffe5166318883c321af18da41ed26934b7e7e75ada1b2d4afbf03c8c563b022b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T01:55:36.537 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:36 smithi090 podman[67950]: 2024-09-21 01:55:36.537220951 +0000 UTC m=+0.347191590 container init ffe5166318883c321af18da41ed26934b7e7e75ada1b2d4afbf03c8c563b022b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T01:55:36.998 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:36 smithi090 podman[67950]: 2024-09-21 01:55:36.540496981 +0000 UTC m=+0.350467622 container start ffe5166318883c321af18da41ed26934b7e7e75ada1b2d4afbf03c8c563b022b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T01:55:36.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:36 smithi090 podman[67950]: 2024-09-21 01:55:36.628854784 +0000 UTC m=+0.438825434 container attach ffe5166318883c321af18da41ed26934b7e7e75ada1b2d4afbf03c8c563b022b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T01:55:36.999 INFO:journalctl@ceph.osd.6.smithi090.stdout:Sep 21 01:55:36 smithi090 podman[67950]: 2024-09-21 01:55:36.723713708 +0000 UTC m=+0.533684349 container died ffe5166318883c321af18da41ed26934b7e7e75ada1b2d4afbf03c8c563b022b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T01:55:37.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.6.service' 2024-09-21T01:55:37.317 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:55:37.317 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T01:55:37.317 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T01:55:37.317 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.7 2024-09-21T01:55:37.657 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 systemd[1]: Stopping Ceph osd.7 for 9b2fd594-77b7-11ef-baf5-efdc52797490... 2024-09-21T01:55:37.657 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.410+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:37.657 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.410+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:37.657 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.410+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:37.657 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.410+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:37.998 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.656+0000 7fc0b1796640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T01:55:37.998 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.656+0000 7fc0b1796640 -1 osd.7 65 *** Got signal Terminated *** 2024-09-21T01:55:37.999 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:37 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:37.656+0000 7fc0b1796640 -1 osd.7 65 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T01:55:38.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:38 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:38.405+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:38.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:38 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:38.405+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:38.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:38 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:38.405+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:38.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:38 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:38.405+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:39.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:39 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:39.433+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:39.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:39 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:39.433+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:39.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:39 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:39.433+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:39.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:39 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:39.433+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:40.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:40.446+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:40.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:40.446+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:40.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:40.446+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:40.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:40.446+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:40.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:40 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:40.446+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.90:6804 osd.4 since back 2024-09-21T01:55:17.465958+0000 front 2024-09-21T01:55:17.466074+0000 (oldest deadline 2024-09-21T01:55:40.365971+0000) 2024-09-21T01:55:41.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:41 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:41.472+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:41.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:41 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:41.472+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:41.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:41 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:41.472+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:41.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:41 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:41.472+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:41.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:41 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:41.472+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.90:6804 osd.4 since back 2024-09-21T01:55:17.465958+0000 front 2024-09-21T01:55:17.466074+0000 (oldest deadline 2024-09-21T01:55:40.365971+0000) 2024-09-21T01:55:43.001 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:42 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:42.521+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6806 osd.0 since back 2024-09-21T01:54:46.461755+0000 front 2024-09-21T01:54:46.461926+0000 (oldest deadline 2024-09-21T01:55:11.761601+0000) 2024-09-21T01:55:43.001 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:42 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:42.521+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6814 osd.1 since back 2024-09-21T01:54:51.762254+0000 front 2024-09-21T01:54:51.762342+0000 (oldest deadline 2024-09-21T01:55:16.462120+0000) 2024-09-21T01:55:43.001 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:42 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:42.521+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6822 osd.2 since back 2024-09-21T01:54:59.963115+0000 front 2024-09-21T01:54:59.963193+0000 (oldest deadline 2024-09-21T01:55:24.663263+0000) 2024-09-21T01:55:43.001 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:42 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:42.521+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.45:6830 osd.3 since back 2024-09-21T01:55:08.064946+0000 front 2024-09-21T01:55:08.065037+0000 (oldest deadline 2024-09-21T01:55:32.765020+0000) 2024-09-21T01:55:43.002 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:42 smithi090 ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7[56321]: 2024-09-21T01:55:42.521+0000 7fc0add81640 -1 osd.7 65 heartbeat_check: no reply from 172.21.15.90:6804 osd.4 since back 2024-09-21T01:55:17.465958+0000 front 2024-09-21T01:55:17.466074+0000 (oldest deadline 2024-09-21T01:55:40.365971+0000) 2024-09-21T01:55:43.344 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:43 smithi090 podman[68055]: 2024-09-21 01:55:43.02518075 +0000 UTC m=+5.485180027 container died effbd3c3de5d72757f6da6ba378dc8d3a81f9b5f7a96acfdb037fb59f7aebf14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7, org.label-schema.build-date=20240731, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T01:55:43.344 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:43 smithi090 podman[68055]: 2024-09-21 01:55:43.344274583 +0000 UTC m=+5.804273860 container cleanup effbd3c3de5d72757f6da6ba378dc8d3a81f9b5f7a96acfdb037fb59f7aebf14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T01:55:43.748 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:43 smithi090 bash[68055]: ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7 2024-09-21T01:55:43.749 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:43 smithi090 podman[68070]: 2024-09-21 01:55:43.562985309 +0000 UTC m=+0.534986929 container remove effbd3c3de5d72757f6da6ba378dc8d3a81f9b5f7a96acfdb037fb59f7aebf14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T01:55:44.300 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:44 smithi090 podman[68161]: 2024-09-21 01:55:43.907031833 +0000 UTC m=+0.015970546 image pull ddce307e1f3263fb077b4f3902e15660050631dad4d2823a20ee3f72bf18d977 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262 2024-09-21T01:55:44.300 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:44 smithi090 podman[68161]: 2024-09-21 01:55:44.005667079 +0000 UTC m=+0.114605832 container create 00446a0e9def72f2de19317894c42b598853a78c2b8b5eb87bdb067e1532c722 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T01:55:44.301 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:44 smithi090 podman[68161]: 2024-09-21 01:55:44.217467502 +0000 UTC m=+0.326406222 container init 00446a0e9def72f2de19317894c42b598853a78c2b8b5eb87bdb067e1532c722 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T01:55:44.301 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:44 smithi090 podman[68161]: 2024-09-21 01:55:44.221430357 +0000 UTC m=+0.330369076 container start 00446a0e9def72f2de19317894c42b598853a78c2b8b5eb87bdb067e1532c722 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T01:55:44.301 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:44 smithi090 podman[68161]: 2024-09-21 01:55:44.300442544 +0000 UTC m=+0.409381264 container attach 00446a0e9def72f2de19317894c42b598853a78c2b8b5eb87bdb067e1532c722 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True) 2024-09-21T01:55:44.704 INFO:journalctl@ceph.osd.7.smithi090.stdout:Sep 21 01:55:44 smithi090 podman[68161]: 2024-09-21 01:55:44.411668977 +0000 UTC m=+0.520607690 container died 00446a0e9def72f2de19317894c42b598853a78c2b8b5eb87bdb067e1532c722 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:dd75f9b101bc316b5d0b4eba020bf68e946d78b16cc3e2ed75055abd1b43a262, name=ceph-9b2fd594-77b7-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240920.212106-debug-1843221, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T01:55:44.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9b2fd594-77b7-11ef-baf5-efdc52797490@osd.7.service' 2024-09-21T01:55:45.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T01:55:45.036 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T01:55:45.036 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T01:55:45.252 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:56:10.715 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T01:56:10.920 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T01:56:24.326 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:56:24.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T01:56:24.387 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T01:56:24.389 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367/remote/smithi045/crash 2024-09-21T01:56:24.390 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/crash -- . 2024-09-21T01:56:24.440 INFO:teuthology.orchestra.run.smithi045.stderr:tar: /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T01:56:24.440 INFO:teuthology.orchestra.run.smithi045.stderr:tar: Error is not recoverable: exiting now 2024-09-21T01:56:24.442 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367/remote/smithi090/crash 2024-09-21T01:56:24.443 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/crash -- . 2024-09-21T01:56:24.482 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T01:56:24.483 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-09-21T01:56:24.484 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T01:56:24.484 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v 'fs.*is degraded' | egrep -v 'filesystem is degraded' | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v 'filesystem is offline' | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'insufficient standby MDS daemons available' | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-21T01:56:24.588 INFO:tasks.cephadm:Compressing logs... 2024-09-21T01:56:24.588 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T01:56:24.590 DEBUG:teuthology.orchestra.run.smithi090:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T01:56:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T01:56:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-21T01:56:24.637 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi045.log 2024-09-21T01:56:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-21T01:56:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi045.log: --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T01:56:24.638 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T01:56:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.bar.smithi045.tspicn.log 2024-09-21T01:56:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T01:56:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.bar.smithi045.tspicn.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mon.smithi045.log 2024-09-21T01:56:24.640 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-21T01:56:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mon.smithi045.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mgr.smithi045.txjudz.log 2024-09-21T01:56:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log 2024-09-21T01:56:24.641 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mgr.smithi045.txjudz.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.foo.smithi045.qpbfpk.log 2024-09-21T01:56:24.642 INFO:teuthology.orchestra.run.smithi045.stderr: 94.4% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi045.log.gz 2024-09-21T01:56:24.642 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-21T01:56:24.642 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.foo.smithi045.qpbfpk.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2024-09-21T01:56:24.642 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T01:56:24.643 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/ceph-client.0.log: /var/log/ceph/cephadm.log: 88.4% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T01:56:24.644 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T01:56:24.644 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mgr.smithi090.zsfvyu.log 2024-09-21T01:56:24.645 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.bar.smithi090.qzpqdi.log 2024-09-21T01:56:24.645 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mgr.smithi090.zsfvyu.log: --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-21T01:56:24.645 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T01:56:24.646 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.bar.smithi090.qzpqdi.log: /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.foo.smithi090.savegz.log 2024-09-21T01:56:24.646 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-21T01:56:24.647 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.foo.smithi090.savegz.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T01:56:24.647 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi090.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.4.log: /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi090.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr: 30.9% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-client.ceph-exporter.smithi090.log.gz 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.6.log: /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mon.smithi090.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.log 2024-09-21T01:56:24.649 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mon.smithi090.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T01:56:24.698 INFO:teuthology.orchestra.run.smithi090.stderr:/var/log/ceph/ceph-client.1.log: /var/log/ceph/cephadm.log: 84.0% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T01:56:24.719 INFO:teuthology.orchestra.run.smithi090.stderr: 93.4% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mgr.smithi090.zsfvyu.log.gz 2024-09-21T01:56:24.752 INFO:teuthology.orchestra.run.smithi090.stderr: 91.7% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T01:56:24.840 INFO:teuthology.orchestra.run.smithi045.stderr: 90.8% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T01:56:24.845 INFO:teuthology.orchestra.run.smithi045.stderr: 91.4% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T01:56:24.861 INFO:teuthology.orchestra.run.smithi045.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2024-09-21T01:56:24.870 INFO:teuthology.orchestra.run.smithi045.stderr: 85.1% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T01:56:24.901 INFO:teuthology.orchestra.run.smithi090.stderr: 93.4% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.foo.smithi090.savegz.log.gz 2024-09-21T01:56:24.935 INFO:teuthology.orchestra.run.smithi090.stderr: 92.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T01:56:24.995 INFO:teuthology.orchestra.run.smithi090.stderr: 95.1% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.bar.smithi090.qzpqdi.log.gz 2024-09-21T01:56:25.075 INFO:teuthology.orchestra.run.smithi090.stderr: 88.5% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T01:56:25.116 INFO:teuthology.orchestra.run.smithi090.stderr: 91.0% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T01:56:25.246 INFO:teuthology.orchestra.run.smithi045.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T01:56:25.657 INFO:teuthology.orchestra.run.smithi045.stderr: 94.8% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.foo.smithi045.qpbfpk.log.gz 2024-09-21T01:56:27.833 INFO:teuthology.orchestra.run.smithi090.stderr: 92.8% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mon.smithi090.log.gz 2024-09-21T01:56:29.926 INFO:teuthology.orchestra.run.smithi045.stderr: 90.1% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mgr.smithi045.txjudz.log.gz 2024-09-21T01:56:43.084 INFO:teuthology.orchestra.run.smithi045.stderr: 90.1% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mon.smithi045.log.gz 2024-09-21T01:56:46.091 INFO:teuthology.orchestra.run.smithi090.stderr: 94.0% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-21T01:56:49.049 INFO:teuthology.orchestra.run.smithi090.stderr: 93.7% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-21T01:57:08.322 INFO:teuthology.orchestra.run.smithi090.stderr: 93.7% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-21T01:57:10.661 INFO:teuthology.orchestra.run.smithi090.stderr: 94.1% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-21T01:57:22.043 INFO:teuthology.orchestra.run.smithi045.stderr: 94.0% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-21T01:57:31.724 INFO:teuthology.orchestra.run.smithi045.stderr: 93.6% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-21T01:58:12.855 INFO:teuthology.orchestra.run.smithi045.stderr: 93.9% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-21T01:58:20.739 INFO:teuthology.orchestra.run.smithi045.stderr: 94.2% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-21T02:02:54.707 INFO:teuthology.orchestra.run.smithi090.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.1.log.gz 2024-09-21T02:02:54.709 INFO:teuthology.orchestra.run.smithi090.stderr: 2024-09-21T02:02:54.709 INFO:teuthology.orchestra.run.smithi090.stderr:real 6m30.104s 2024-09-21T02:02:54.709 INFO:teuthology.orchestra.run.smithi090.stderr:user 5m48.755s 2024-09-21T02:02:54.709 INFO:teuthology.orchestra.run.smithi090.stderr:sys 0m14.311s 2024-09-21T02:08:03.368 INFO:teuthology.orchestra.run.smithi045.stderr: 94.5% -- replaced with /var/log/ceph/9b2fd594-77b7-11ef-baf5-efdc52797490/ceph-mds.bar.smithi045.tspicn.log.gz 2024-09-21T02:08:03.394 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-09-21T02:08:03.394 INFO:teuthology.orchestra.run.smithi045.stderr:real 11m38.789s 2024-09-21T02:08:03.394 INFO:teuthology.orchestra.run.smithi045.stderr:user 9m25.209s 2024-09-21T02:08:03.395 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m23.859s 2024-09-21T02:08:03.395 INFO:tasks.cephadm:Archiving logs... 2024-09-21T02:08:03.396 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367/remote/smithi045/log 2024-09-21T02:08:03.396 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T02:09:16.618 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367/remote/smithi090/log 2024-09-21T02:09:16.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T02:10:17.303 INFO:tasks.cephadm:Removing cluster... 2024-09-21T02:10:17.303 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 --force 2024-09-21T02:10:18.028 INFO:teuthology.orchestra.run.smithi045.stdout:Deleting cluster with fsid: 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T02:10:19.016 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9b2fd594-77b7-11ef-baf5-efdc52797490 --force 2024-09-21T02:10:19.647 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 9b2fd594-77b7-11ef-baf5-efdc52797490 2024-09-21T02:10:19.932 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T02:10:19.932 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T02:10:19.950 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T02:10:19.966 INFO:tasks.cephadm:Teardown complete 2024-09-21T02:10:19.966 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T02:10:19.981 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T02:10:19.981 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T02:10:19.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T02:10:20.093 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T02:10:20.103 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T02:10:20.103 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:10:20.106 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T02:10:20.120 INFO:teuthology.orchestra.run.smithi045.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:10:20.121 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-09-21T02:10:20.155 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:10:20.155 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-09-21T02:10:20.155 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv01.front.sepia.ceph.com 3 6 377 63 +59us[ +125us] +/- 40ms 2024-09-21T02:10:20.155 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 62 -712us[ -712us] +/- 60ms 2024-09-21T02:10:20.155 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +830us[ +830us] +/- 71ms 2024-09-21T02:10:20.155 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:10:20.221 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T02:10:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-09-21T02:10:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv01.front.sepia.ceph.com 3 6 377 62 +41us[ +52us] +/- 40ms 2024-09-21T02:10:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 63 -716us[ -705us] +/- 60ms 2024-09-21T02:10:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +814us[ +814us] +/- 71ms 2024-09-21T02:10:20.222 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T02:10:20.229 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T02:10:20.238 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T02:10:20.239 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T02:10:20.279 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T02:10:20.428 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T02:10:20.522 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T02:10:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:10:20.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T02:10:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:10:20.583 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T02:10:20.592 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T02:10:20.677 INFO:teuthology.task.internal:Duration was 3400.135734 seconds 2024-09-21T02:10:20.678 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T02:10:20.688 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T02:10:20.688 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T02:10:20.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T02:10:20.750 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:10:20.767 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T02:10:21.273 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T02:10:21.273 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-09-21T02:10:21.273 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T02:10:21.307 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-09-21T02:10:21.307 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T02:10:21.331 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T02:10:21.331 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T02:10:21.349 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T02:10:21.398 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T02:10:21.398 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T02:10:21.434 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T02:10:21.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T02:10:21.870 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T02:10:21.870 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T02:10:21.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T02:10:21.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T02:10:21.955 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:10:21.958 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:10:21.994 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-09-21T02:10:22.023 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-09-21T02:10:22.059 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:10:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:10:22.074 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T02:10:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T02:10:22.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T02:10:22.127 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T02:10:22.128 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367/remote/smithi045 2024-09-21T02:10:22.128 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T02:10:22.169 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-09-20_23:48:22-fs-wip-pdonnell-testing-20240920.212106-debug-distro-default-smithi/7913367/remote/smithi090 2024-09-21T02:10:22.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T02:10:22.205 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T02:10:22.205 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T02:10:22.208 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T02:10:22.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T02:10:22.266 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T02:10:22.266 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T02:10:22.284 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T02:10:22.284 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T02:10:22.286 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T02:10:22.300 INFO:teuthology.orchestra.run.smithi045.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 02:10 /home/ubuntu/cephtest 2024-09-21T02:10:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 02:10 /home/ubuntu/cephtest 2024-09-21T02:10:22.321 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T02:10:22.332 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T02:10:22.362 INFO:teuthology.run:Summary data: description: fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench conf/{client mds mgr mon osd} distro/single-container-host overrides/{ignorelist_health pg_health}} duration: 3400.135734319687 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-09-21T02:10:22.363 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T02:10:22.432 INFO:teuthology.run:pass