2024-09-13T15:14:58.229 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T15:14:58.246 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T15:14:58.342 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734 branch: wip-rishabh-testing-20240910.095512-reef description: fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench conf/{client mds mgr mon osd} distro/single-container-host overrides/{ignorelist_health pg_health}} email: null first_in_suite: false job_id: '7903734' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240910.095512-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd op complaint time: 180 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 50bbe7f395997dcd871813949c49addbbf630361 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 50bbe7f395997dcd871813949c49addbbf630361 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-rishabh-testing-20240910.095512-reef sha1: 50bbe7f395997dcd871813949c49addbbf630361 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 1152 sha1: 50bbe7f395997dcd871813949c49addbbf630361 sleep_before_teardown: 0 subset: 984/1024 suite: fs suite_branch: wip-rishabh-testing-20240910.095512-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 50bbe7f395997dcd871813949c49addbbf630361 targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo1k+fyHWaOQOdfzGJC4zSiLgzbSaJiLMq1s+8m6cikcoYec7JTm2JcUSDVJDitDLYLtN6Jka2OxeylQXnbCQ0= smithi175.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpVhnk3pkNUnzFUuBbnxdbtBdrHPeTsxS7WjBiGnearapi9HOxwhZKJMigmb+CRNMNtQNBekSCRSE/rCON1AmQ= tasks: - install: null - cephadm: roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create foo - ceph fs volume create bar - fs.ready: timeout: 300 - ceph-fuse: client.0: cephfs_name: foo - ceph-fuse: client.1: cephfs_name: bar - workunit: clients: all: - suites/dbench.sh - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_14:36:41 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T15:14:58.342 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa; will attempt to use it 2024-09-13T15:14:58.343 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa/tasks 2024-09-13T15:14:58.343 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T15:14:58.345 INFO:teuthology.task.internal:Checking packages... 2024-09-13T15:14:58.366 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '50bbe7f395997dcd871813949c49addbbf630361' 2024-09-13T15:14:58.366 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T15:14:58.367 INFO:teuthology.packaging:ref: None 2024-09-13T15:14:58.367 INFO:teuthology.packaging:tag: None 2024-09-13T15:14:58.367 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240910.095512-reef 2024-09-13T15:14:58.367 INFO:teuthology.packaging:sha1: 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:14:58.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240910.095512-reef 2024-09-13T15:14:58.522 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-850.g50bbe7f3 2024-09-13T15:14:58.524 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T15:14:58.531 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T15:14:58.531 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T15:14:58.536 INFO:teuthology.task.internal:Saving configuration 2024-09-13T15:14:58.549 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T15:14:58.556 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T15:14:58.589 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 15:09:13.780299', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPo1k+fyHWaOQOdfzGJC4zSiLgzbSaJiLMq1s+8m6cikcoYec7JTm2JcUSDVJDitDLYLtN6Jka2OxeylQXnbCQ0='} 2024-09-13T15:14:58.619 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 15:09:13.778324', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpVhnk3pkNUnzFUuBbnxdbtBdrHPeTsxS7WjBiGnearapi9HOxwhZKJMigmb+CRNMNtQNBekSCRSE/rCON1AmQ='} 2024-09-13T15:14:58.619 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T15:14:58.625 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-13T15:14:58.625 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-13T15:14:58.625 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T15:14:58.709 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fadcfb97be0>, signals=[15]) 2024-09-13T15:14:58.709 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T15:14:58.715 INFO:teuthology.task.internal:Opening connections... 2024-09-13T15:14:58.715 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2024-09-13T15:14:58.716 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:14:58.804 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2024-09-13T15:14:58.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:14:58.902 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T15:14:58.909 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2024-09-13T15:14:58.927 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2024-09-13T15:14:58.927 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2024-09-13T15:14:58.984 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="9" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="9" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el9" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:LOGO="fedora-logo-icon" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T15:14:58.985 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T15:14:58.986 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T15:14:58.986 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2024-09-13T15:14:59.033 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2024-09-13T15:14:59.052 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2024-09-13T15:14:59.052 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2024-09-13T15:14:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="CentOS Stream" 2024-09-13T15:14:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="9" 2024-09-13T15:14:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:ID="centos" 2024-09-13T15:14:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="rhel fedora" 2024-09-13T15:14:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="9" 2024-09-13T15:14:59.110 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el9" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:LOGO="fedora-logo-icon" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://centos.org/" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T15:14:59.111 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T15:14:59.111 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2024-09-13T15:14:59.141 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T15:14:59.150 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T15:14:59.188 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T15:14:59.188 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2024-09-13T15:14:59.191 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2024-09-13T15:14:59.207 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T15:14:59.213 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T15:14:59.213 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2024-09-13T15:14:59.248 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2024-09-13T15:14:59.354 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T15:14:59.428 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T15:14:59.442 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T15:14:59.442 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T15:14:59.443 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T15:14:59.443 DEBUG:teuthology.orchestra.run.smithi175:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T15:14:59.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:14:59.460 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-09-13T15:14:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:14:59.461 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-09-13T15:14:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-503.el9.x86_64 2024-09-13T15:14:59.519 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.14.0-503.el9.x86_64 2024-09-13T15:14:59.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum install -y kernel 2024-09-13T15:14:59.520 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-503.el9.x86_64 2024-09-13T15:14:59.520 INFO:teuthology.task.kernel:Running kernel on smithi175: 5.14.0-503.el9.x86_64 2024-09-13T15:14:59.520 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2024-09-13T15:15:02.283 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.2 MB 00:00 2024-09-13T15:15:03.144 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-13T15:15:04.043 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-13T15:15:07.928 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 20 MB 00:03 2024-09-13T15:15:08.495 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-13T15:15:09.946 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 19 kB 00:00 2024-09-13T15:15:11.393 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-13T15:15:12.470 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.5 MB 00:00 2024-09-13T15:15:13.913 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 19 kB 00:00 2024-09-13T15:15:15.150 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-13T15:15:18.425 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 1.0 kB/s | 1.7 kB 00:01 2024-09-13T15:15:20.481 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T15:15:20.482 INFO:teuthology.orchestra.run.smithi090.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T15:15:20.555 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-13T15:15:20.596 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:15:20.610 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:15:20.610 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T15:15:20.611 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 79 M 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 126 M 2024-09-13T15:15:20.612 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:15:20.743 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 205 kB/s | 23 kB 00:00 2024-09-13T15:15:21.836 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-13T15:15:22.444 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 17 MB/s | 29 MB 00:01 2024-09-13T15:15:22.553 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-13T15:15:22.553 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-13T15:15:22.634 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-13T15:15:22.643 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-13T15:15:22.644 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-13T15:15:22.644 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:15:22.644 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:15:22.644 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:15:22.644 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2024-09-13T15:15:22.644 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:15:22.645 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 79 M 2024-09-13T15:15:22.645 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 126 M 2024-09-13T15:15:22.645 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:15:22.819 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-13T15:15:22.820 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:15:22.820 INFO:teuthology.orchestra.run.smithi090.stdout:Total 36 MB/s | 79 MB 00:02 2024-09-13T15:15:22.998 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 233 kB/s | 23 kB 00:00 2024-09-13T15:15:23.499 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:15:23.874 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:15:23.874 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:15:24.780 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:15:24.781 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:15:26.539 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.3 MB/s | 16 MB 00:03 2024-09-13T15:15:27.279 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:15:27.806 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 7.0 MB/s | 35 MB 00:04 2024-09-13T15:15:27.968 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T15:15:28.050 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T15:15:28.231 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 5.5 MB/s | 29 MB 00:05 2024-09-13T15:15:28.232 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:15:28.232 INFO:teuthology.orchestra.run.smithi175.stdout:Total 14 MB/s | 79 MB 00:05 2024-09-13T15:15:28.920 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:15:29.265 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:15:29.265 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:15:30.252 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:15:30.253 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:15:30.839 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T15:15:31.012 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T15:15:32.700 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:15:33.405 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-13T15:15:33.489 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T15:15:35.851 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T15:15:35.889 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:15:36.321 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T15:15:36.451 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T15:15:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:15:41.295 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T15:15:41.343 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:15:46.180 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T15:16:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T15:16:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T15:16:22.049 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:16:22.050 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:16:22.444 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T15:16:22.551 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T15:16:22.552 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T15:16:22.552 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T15:16:22.552 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T15:16:22.552 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T15:16:22.552 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-09-13T15:16:22.552 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T15:16:22.552 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T15:16:22.612 INFO:teuthology.orchestra.run.smithi090.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T15:16:22.613 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T15:16:22.613 DEBUG:teuthology.orchestra.run.smithi090:> sudo rpm -qi grub2-tools 2024-09-13T15:16:22.644 INFO:teuthology.orchestra.run.smithi090.stdout:Name : grub2-tools 2024-09-13T15:16:22.644 INFO:teuthology.orchestra.run.smithi090.stdout:Epoch : 1 2024-09-13T15:16:22.644 INFO:teuthology.orchestra.run.smithi090.stdout:Version : 2.06 2024-09-13T15:16:22.644 INFO:teuthology.orchestra.run.smithi090.stdout:Release : 82.el9 2024-09-13T15:16:22.644 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: x86_64 2024-09-13T15:16:22.644 INFO:teuthology.orchestra.run.smithi090.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Group : Unspecified 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Size : 8274670 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:License : GPLv3+ 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Packager : builder@centos.org 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Vendor : CentOS 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Summary : Support tools for GRUB. 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout:Description : 2024-09-13T15:16:22.645 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:16:22.646 INFO:teuthology.orchestra.run.smithi090.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T15:16:22.646 INFO:teuthology.orchestra.run.smithi090.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T15:16:22.646 INFO:teuthology.orchestra.run.smithi090.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T15:16:22.646 INFO:teuthology.orchestra.run.smithi090.stdout:hardware devices. 2024-09-13T15:16:22.646 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:16:22.646 INFO:teuthology.orchestra.run.smithi090.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T15:16:22.647 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T15:16:22.647 INFO:teuthology.task.kernel:Updating grub on smithi090 to boot 5.14.0-505.el9.x86_64 2024-09-13T15:16:22.647 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T15:16:23.274 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2024-09-13T15:16:24.505 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:24.545 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:24.899 INFO:teuthology.orchestra.run.smithi090.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T15:16:24.928 INFO:teuthology.orchestra.run.smithi090.stderr:done 2024-09-13T15:16:24.930 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2024-09-13T15:16:24.947 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:24.948 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-13T15:16:24.948 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-13T15:16:24.948 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-13T15:16:24.950 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.WIjRUl4viA 2024-09-13T15:16:24.951 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WIjRUl4viA 2024-09-13T15:16:25.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.WIjRUl4viA 2024-09-13T15:16:25.171 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.WIjRUl4viA is 6KB 2024-09-13T15:16:25.191 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.WIjRUl4viA 2024-09-13T15:16:25.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T15:16:25.279 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T15:16:25.280 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T15:16:25.280 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T15:16:25.280 INFO:teuthology.orchestra.run.smithi090.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T15:16:25.281 DEBUG:teuthology.orchestra.run.smithi090:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T15:16:25.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:16:25.873 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:16:26.206 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T15:16:26.289 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T15:16:26.289 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T15:16:26.289 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-13T15:16:26.290 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T15:16:26.290 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T15:16:26.290 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-09-13T15:16:26.290 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi175.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T15:16:26.290 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-09-13T15:16:26.349 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-13T15:16:26.349 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-13T15:16:26.350 DEBUG:teuthology.orchestra.run.smithi175:> sudo rpm -qi grub2-tools 2024-09-13T15:16:26.390 INFO:teuthology.orchestra.run.smithi175.stdout:Name : grub2-tools 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Epoch : 1 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Version : 2.06 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Release : 82.el9 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Architecture: x86_64 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Group : Unspecified 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Size : 8274670 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:License : GPLv3+ 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T15:16:26.391 INFO:teuthology.orchestra.run.smithi175.stdout:Packager : builder@centos.org 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:Vendor : CentOS 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:Summary : Support tools for GRUB. 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:Description : 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:hardware devices. 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:16:26.392 INFO:teuthology.orchestra.run.smithi175.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T15:16:26.393 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T15:16:26.393 INFO:teuthology.task.kernel:Updating grub on smithi175 to boot 5.14.0-505.el9.x86_64 2024-09-13T15:16:26.393 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T15:16:27.001 INFO:teuthology.orchestra.run.smithi175.stderr:Generating grub configuration file ... 2024-09-13T15:16:28.341 INFO:teuthology.orchestra.run.smithi175.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T15:16:28.361 INFO:teuthology.orchestra.run.smithi175.stderr:done 2024-09-13T15:16:28.369 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2024-09-13T15:16:28.385 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.gS7P3NndUj 2024-09-13T15:16:28.385 DEBUG:teuthology.orchestra.run.smithi175:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gS7P3NndUj 2024-09-13T15:16:28.464 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.gS7P3NndUj 2024-09-13T15:16:28.607 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.gS7P3NndUj is 6KB 2024-09-13T15:16:28.628 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.gS7P3NndUj 2024-09-13T15:16:28.643 DEBUG:teuthology.orchestra.run.smithi175:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T15:16:28.707 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T15:16:28.707 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T15:16:28.707 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T15:16:28.708 INFO:teuthology.orchestra.run.smithi175.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-13T15:16:28.709 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-13T15:16:28.895 DEBUG:teuthology.orchestra.run.smithi175:> sudo shutdown -r now 2024-09-13T15:16:55.470 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-09-13T15:16:55.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-13T15:16:55.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:16:58.900 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-09-13T15:16:58.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-13T15:16:58.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:17:30.045 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-09-13T15:17:39.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-13T15:17:39.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:17:42.145 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2024-09-13T15:17:54.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-13T15:17:54.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:17:58.916 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T15:18:07.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-13T15:18:07.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:18:08.428 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-09-13T15:18:08.811 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-13T15:18:08.811 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-505.el9.x86_64"... 2024-09-13T15:18:08.812 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-09-13T15:18:08.826 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-505.el9.x86_64 2024-09-13T15:18:08.827 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T15:18:08.827 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T15:18:08.827 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T15:18:09.827 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-09-13T15:18:09.827 DEBUG:teuthology.orchestra.run.smithi175:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T15:18:09.913 INFO:teuthology.orchestra.run.smithi175.stdout:ttyS1 2024-09-13T15:18:09.938 DEBUG:teuthology.parallel:result is None 2024-09-13T15:18:26.328 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-09-13T15:18:27.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-13T15:18:27.118 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-505.el9.x86_64"... 2024-09-13T15:18:27.118 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2024-09-13T15:18:27.132 INFO:teuthology.orchestra.run.smithi090.stdout:5.14.0-505.el9.x86_64 2024-09-13T15:18:27.132 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-13T15:18:27.132 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T15:18:27.133 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T15:18:28.133 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-09-13T15:18:28.134 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T15:18:28.277 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2024-09-13T15:18:28.296 DEBUG:teuthology.parallel:result is None 2024-09-13T15:18:28.296 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T15:18:28.304 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T15:18:28.304 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T15:18:28.307 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T15:18:28.327 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T15:18:28.333 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T15:18:28.343 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T15:18:28.343 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T15:18:28.358 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T15:18:28.407 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T15:18:28.413 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T15:18:28.413 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T15:18:28.424 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T15:18:28.474 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:28.484 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:28.486 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:28.495 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T15:18:28.497 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T15:18:28.504 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T15:18:28.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T15:18:28.506 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T15:18:28.556 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T15:18:28.565 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T15:18:28.566 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T15:18:28.569 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T15:18:28.613 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T15:18:28.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T15:18:28.715 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T15:18:28.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T15:18:28.800 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:18:28.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T15:18:28.877 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T15:18:28.930 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T15:18:28.977 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T15:18:29.066 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T15:18:29.102 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:18:29.102 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T15:18:29.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2024-09-13T15:18:29.170 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2024-09-13T15:18:29.218 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:18:29.261 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T15:18:29.552 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T15:18:29.560 INFO:teuthology.task.internal:Starting timer... 2024-09-13T15:18:29.560 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T15:18:29.570 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T15:18:29.579 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-13T15:18:29.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo service auditd rotate 2024-09-13T15:18:29.653 INFO:teuthology.orchestra.run.smithi090.stdout:Rotating logs: 2024-09-13T15:18:29.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2024-09-13T15:18:29.719 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: 2024-09-13T15:18:29.721 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T15:18:29.721 DEBUG:teuthology.orchestra.run.smithi090:> /usr/sbin/getenforce 2024-09-13T15:18:29.747 INFO:teuthology.orchestra.run.smithi090.stdout:Permissive 2024-09-13T15:18:29.748 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2024-09-13T15:18:29.771 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2024-09-13T15:18:29.771 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi090.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi175.front.sepia.ceph.com': 'permissive'} 2024-09-13T15:18:29.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T15:18:29.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:18:29.810 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T15:18:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:18:29.837 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T15:18:29.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo /usr/sbin/setenforce permissive 2024-09-13T15:18:29.878 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2024-09-13T15:18:29.911 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T15:18:29.920 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T15:18:29.932 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T15:18:29.933 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi175.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T15:22:12.252 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi175.front.sepia.ceph.com')] 2024-09-13T15:22:12.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-13T15:22:12.255 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:22:12.342 DEBUG:teuthology.orchestra.run.smithi090:> true 2024-09-13T15:22:12.424 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2024-09-13T15:22:12.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-13T15:22:12.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T15:22:12.505 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-09-13T15:22:12.583 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-09-13T15:22:12.583 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T15:22:12.593 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T15:22:12.593 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T15:22:12.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:22:12.596 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T15:22:12.596 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T15:22:12.623 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T15:22:12.637 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T15:22:12.651 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T15:22:12.661 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: ntpd: command not found 2024-09-13T15:22:12.665 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T15:22:12.672 INFO:teuthology.orchestra.run.smithi090.stdout:506 Cannot talk to daemon 2024-09-13T15:22:12.686 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T15:22:12.689 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2024-09-13T15:22:12.700 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2024-09-13T15:22:12.701 INFO:teuthology.orchestra.run.smithi090.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T15:22:12.714 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T15:22:12.728 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T15:22:12.758 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:22:12.760 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:22:12.760 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-09-13T15:22:12.761 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.761 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.761 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.761 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.787 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2024-09-13T15:22:12.790 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T15:22:12.790 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-09-13T15:22:12.790 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.790 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.790 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.790 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T15:22:12.791 INFO:teuthology.run_tasks:Running task install... 2024-09-13T15:22:12.798 DEBUG:teuthology.task.install:project ceph 2024-09-13T15:22:12.798 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}} 2024-09-13T15:22:12.799 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'} 2024-09-13T15:22:12.799 INFO:teuthology.task.install:Using flavor: default 2024-09-13T15:22:12.804 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T15:22:12.804 INFO:teuthology.task.install:extra packages: [] 2024-09-13T15:22:12.805 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'tag': None, 'wait_for_package': False} 2024-09-13T15:22:12.805 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:22:12.807 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'tag': None, 'wait_for_package': False} 2024-09-13T15:22:12.807 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:22:12.954 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/ 2024-09-13T15:22:12.954 INFO:teuthology.task.install.rpm:Package version is 18.2.4-850.g50bbe7f3 2024-09-13T15:22:12.971 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/ 2024-09-13T15:22:12.971 INFO:teuthology.task.install.rpm:Package version is 18.2.4-850.g50bbe7f3 2024-09-13T15:22:13.076 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T15:22:13.076 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:22:13.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T15:22:13.087 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T15:22:13.088 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:22:13.088 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T15:22:13.101 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T15:22:13.101 DEBUG:teuthology.orchestra.run.smithi090:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/50bbe7f395997dcd871813949c49addbbf630361/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T15:22:13.114 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-13T15:22:13.114 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/50bbe7f395997dcd871813949c49addbbf630361/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T15:22:13.167 DEBUG:teuthology.orchestra.run.smithi090:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T15:22:13.180 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T15:22:13.242 DEBUG:teuthology.orchestra.run.smithi090:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T15:22:13.254 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T15:22:13.306 INFO:teuthology.orchestra.run.smithi090.stdout:check_obsoletes = 1 2024-09-13T15:22:13.307 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum clean all 2024-09-13T15:22:13.319 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2024-09-13T15:22:13.320 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2024-09-13T15:22:13.540 INFO:teuthology.orchestra.run.smithi090.stdout:45 files removed 2024-09-13T15:22:13.553 INFO:teuthology.orchestra.run.smithi175.stdout:45 files removed 2024-09-13T15:22:13.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-radosgw 2024-09-13T15:22:13.580 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw 2024-09-13T15:22:14.122 INFO:teuthology.orchestra.run.smithi090.stdout:ceph packages for x86_64 285 kB/s | 77 kB 00:00 2024-09-13T15:22:14.137 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 282 kB/s | 77 kB 00:00 2024-09-13T15:22:14.406 INFO:teuthology.orchestra.run.smithi090.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-13T15:22:14.421 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 48 kB/s | 11 kB 00:00 2024-09-13T15:22:14.682 INFO:teuthology.orchestra.run.smithi090.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-13T15:22:14.697 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-13T15:22:15.214 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-09-13T15:22:16.657 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-13T15:22:17.132 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-13T15:22:18.459 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - AppStream 30 MB/s | 20 MB 00:00 2024-09-13T15:22:22.921 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-13T15:22:23.124 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.5 MB 00:02 2024-09-13T15:22:24.389 INFO:teuthology.orchestra.run.smithi090.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 19 kB 00:00 2024-09-13T15:22:24.741 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-13T15:22:26.114 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 18 MB/s | 23 MB 00:01 2024-09-13T15:22:26.226 INFO:teuthology.orchestra.run.smithi090.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-13T15:22:31.553 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-13T15:22:31.750 INFO:teuthology.orchestra.run.smithi090.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-13T15:22:33.920 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 7.8 M 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.3 M 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 5.1 M 2024-09-13T15:22:33.928 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 18 M 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 25 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 712 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 477 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T15:22:33.929 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.5 M 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 45 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 129 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 162 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 319 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 301 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 99 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:22:33.930 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:22:33.931 INFO:teuthology.orchestra.run.smithi175.stdout:Install 28 Packages 2024-09-13T15:22:33.931 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 2 Packages 2024-09-13T15:22:33.931 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:33.931 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 53 M 2024-09-13T15:22:33.931 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:22:34.143 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 7.8 M 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrading: 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.3 M 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T15:22:34.158 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 5.1 M 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 18 M 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 25 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 712 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 477 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T15:22:34.159 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2 x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.5 M 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 45 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 129 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 162 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 319 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 301 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 99 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T15:22:34.160 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout:Install 28 Packages 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout:Upgrade 2 Packages 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 53 M 2024-09-13T15:22:34.161 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:22:34.804 INFO:teuthology.orchestra.run.smithi175.stdout:(1/30): ceph-base-18.2.4-850.g50bbe7f3.el9.x86_ 9.2 MB/s | 5.1 MB 00:00 2024-09-13T15:22:34.904 INFO:teuthology.orchestra.run.smithi175.stdout:(2/30): ceph-radosgw-18.2.4-850.g50bbe7f3.el9.x 12 MB/s | 7.8 MB 00:00 2024-09-13T15:22:34.921 INFO:teuthology.orchestra.run.smithi175.stdout:(3/30): ceph-selinux-18.2.4-850.g50bbe7f3.el9.x 215 kB/s | 25 kB 00:00 2024-09-13T15:22:34.980 INFO:teuthology.orchestra.run.smithi175.stdout:(4/30): libcephfs2-18.2.4-850.g50bbe7f3.el9.x86 9.3 MB/s | 712 kB 00:00 2024-09-13T15:22:35.005 INFO:teuthology.orchestra.run.smithi175.stdout:(5/30): libradosstriper1-18.2.4-850.g50bbe7f3.e 5.6 MB/s | 477 kB 00:00 2024-09-13T15:22:35.280 INFO:teuthology.orchestra.run.smithi175.stdout:(6/30): librgw2-18.2.4-850.g50bbe7f3.el9.x86_64 15 MB/s | 4.5 MB 00:00 2024-09-13T15:22:35.464 INFO:teuthology.orchestra.run.smithi175.stdout:(7/30): ceph-common-18.2.4-850.g50bbe7f3.el9.x8 15 MB/s | 18 MB 00:01 2024-09-13T15:22:35.489 INFO:teuthology.orchestra.run.smithi175.stdout:(8/30): python3-ceph-argparse-18.2.4-850.g50bbe 93 kB/s | 45 kB 00:00 2024-09-13T15:22:35.510 INFO:teuthology.orchestra.run.smithi090.stdout:(1/30): ceph-radosgw-18.2.4-850.g50bbe7f3.el9.x 9.0 MB/s | 7.8 MB 00:00 2024-09-13T15:22:35.514 INFO:teuthology.orchestra.run.smithi175.stdout:(9/30): python3-ceph-common-18.2.4-850.g50bbe7f 552 kB/s | 129 kB 00:00 2024-09-13T15:22:35.539 INFO:teuthology.orchestra.run.smithi175.stdout:(10/30): python3-cephfs-18.2.4-850.g50bbe7f3.el 2.1 MB/s | 162 kB 00:00 2024-09-13T15:22:35.565 INFO:teuthology.orchestra.run.smithi175.stdout:(11/30): python3-rados-18.2.4-850.g50bbe7f3.el9 4.1 MB/s | 319 kB 00:00 2024-09-13T15:22:35.590 INFO:teuthology.orchestra.run.smithi175.stdout:(12/30): python3-rbd-18.2.4-850.g50bbe7f3.el9.x 3.9 MB/s | 301 kB 00:00 2024-09-13T15:22:35.602 INFO:teuthology.orchestra.run.smithi090.stdout:(2/30): ceph-base-18.2.4-850.g50bbe7f3.el9.x86_ 5.4 MB/s | 5.1 MB 00:00 2024-09-13T15:22:35.615 INFO:teuthology.orchestra.run.smithi175.stdout:(13/30): python3-rgw-18.2.4-850.g50bbe7f3.el9.x 1.3 MB/s | 99 kB 00:00 2024-09-13T15:22:35.628 INFO:teuthology.orchestra.run.smithi090.stdout:(3/30): ceph-selinux-18.2.4-850.g50bbe7f3.el9.x 213 kB/s | 25 kB 00:00 2024-09-13T15:22:35.687 INFO:teuthology.orchestra.run.smithi090.stdout:(4/30): libradosstriper1-18.2.4-850.g50bbe7f3.e 8.0 MB/s | 477 kB 00:00 2024-09-13T15:22:35.721 INFO:teuthology.orchestra.run.smithi090.stdout:(5/30): libcephfs2-18.2.4-850.g50bbe7f3.el9.x86 5.9 MB/s | 712 kB 00:00 2024-09-13T15:22:35.724 INFO:teuthology.orchestra.run.smithi175.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-13T15:22:35.757 INFO:teuthology.orchestra.run.smithi175.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 556 kB/s | 107 kB 00:00 2024-09-13T15:22:35.779 INFO:teuthology.orchestra.run.smithi090.stdout:(6/30): python3-ceph-argparse-18.2.4-850.g50bbe 771 kB/s | 45 kB 00:00 2024-09-13T15:22:35.791 INFO:teuthology.orchestra.run.smithi175.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 798 kB/s | 160 kB 00:00 2024-09-13T15:22:35.838 INFO:teuthology.orchestra.run.smithi090.stdout:(7/30): python3-ceph-common-18.2.4-850.g50bbe7f 2.2 MB/s | 129 kB 00:00 2024-09-13T15:22:35.841 INFO:teuthology.orchestra.run.smithi175.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 842 kB/s | 42 kB 00:00 2024-09-13T15:22:35.867 INFO:teuthology.orchestra.run.smithi175.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-13T15:22:35.913 INFO:teuthology.orchestra.run.smithi090.stdout:(8/30): librgw2-18.2.4-850.g50bbe7f3.el9.x86_64 20 MB/s | 4.5 MB 00:00 2024-09-13T15:22:35.917 INFO:teuthology.orchestra.run.smithi175.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 1.4 MB/s | 111 kB 00:00 2024-09-13T15:22:35.939 INFO:teuthology.orchestra.run.smithi090.stdout:(9/30): python3-cephfs-18.2.4-850.g50bbe7f3.el9 1.6 MB/s | 162 kB 00:00 2024-09-13T15:22:35.942 INFO:teuthology.orchestra.run.smithi175.stdout:(20/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 662 kB 00:00 2024-09-13T15:22:35.964 INFO:teuthology.orchestra.run.smithi090.stdout:(10/30): python3-rados-18.2.4-850.g50bbe7f3.el9 6.2 MB/s | 319 kB 00:00 2024-09-13T15:22:35.984 INFO:teuthology.orchestra.run.smithi175.stdout:(21/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.6 MB/s | 308 kB 00:00 2024-09-13T15:22:35.989 INFO:teuthology.orchestra.run.smithi090.stdout:(11/30): python3-rbd-18.2.4-850.g50bbe7f3.el9.x 5.8 MB/s | 301 kB 00:00 2024-09-13T15:22:36.015 INFO:teuthology.orchestra.run.smithi090.stdout:(12/30): python3-rgw-18.2.4-850.g50bbe7f3.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-13T15:22:36.018 INFO:teuthology.orchestra.run.smithi175.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-09-13T15:22:36.043 INFO:teuthology.orchestra.run.smithi175.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 856 kB/s | 49 kB 00:00 2024-09-13T15:22:36.068 INFO:teuthology.orchestra.run.smithi175.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T15:22:36.110 INFO:teuthology.orchestra.run.smithi175.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-13T15:22:36.152 INFO:teuthology.orchestra.run.smithi175.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-13T15:22:36.182 INFO:teuthology.orchestra.run.smithi090.stdout:(13/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 961 kB/s | 160 kB 00:00 2024-09-13T15:22:36.215 INFO:teuthology.orchestra.run.smithi090.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 474 kB/s | 107 kB 00:00 2024-09-13T15:22:36.233 INFO:teuthology.orchestra.run.smithi090.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-09-13T15:22:36.261 INFO:teuthology.orchestra.run.smithi175.stdout:(27/30): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-13T15:22:36.311 INFO:teuthology.orchestra.run.smithi175.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-09-13T15:22:36.333 INFO:teuthology.orchestra.run.smithi090.stdout:(16/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-09-13T15:22:36.358 INFO:teuthology.orchestra.run.smithi090.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-13T15:22:36.392 INFO:teuthology.orchestra.run.smithi090.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 720 kB/s | 42 kB 00:00 2024-09-13T15:22:36.394 INFO:teuthology.orchestra.run.smithi175.stdout:(29/30): librados2-18.2.4-850.g50bbe7f3.el9.x86 14 MB/s | 3.3 MB 00:00 2024-09-13T15:22:36.434 INFO:teuthology.orchestra.run.smithi090.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 1.4 MB/s | 111 kB 00:00 2024-09-13T15:22:36.470 INFO:teuthology.orchestra.run.smithi175.stdout:(30/30): librbd1-18.2.4-850.g50bbe7f3.el9.x86_6 14 MB/s | 3.0 MB 00:00 2024-09-13T15:22:36.475 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:22:36.475 INFO:teuthology.orchestra.run.smithi175.stdout:Total 21 MB/s | 53 MB 00:02 2024-09-13T15:22:36.484 INFO:teuthology.orchestra.run.smithi090.stdout:(20/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-13T15:22:36.510 INFO:teuthology.orchestra.run.smithi090.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-13T15:22:36.535 INFO:teuthology.orchestra.run.smithi090.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-13T15:22:36.553 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:22:36.569 INFO:teuthology.orchestra.run.smithi090.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-13T15:22:36.588 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:22:36.589 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:22:36.644 INFO:teuthology.orchestra.run.smithi090.stdout:(24/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-13T15:22:36.686 INFO:teuthology.orchestra.run.smithi090.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-13T15:22:36.763 INFO:teuthology.orchestra.run.smithi090.stdout:(26/30): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-13T15:22:36.855 INFO:teuthology.orchestra.run.smithi090.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 9.4 MB/s | 1.6 MB 00:00 2024-09-13T15:22:37.055 INFO:teuthology.orchestra.run.smithi090.stdout:(28/30): librbd1-18.2.4-850.g50bbe7f3.el9.x86_6 15 MB/s | 3.0 MB 00:00 2024-09-13T15:22:37.164 INFO:teuthology.orchestra.run.smithi090.stdout:(29/30): librados2-18.2.4-850.g50bbe7f3.el9.x86 8.3 MB/s | 3.3 MB 00:00 2024-09-13T15:22:37.229 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:22:37.230 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:22:37.839 INFO:teuthology.orchestra.run.smithi090.stdout:(30/30): ceph-common-18.2.4-850.g50bbe7f3.el9.x 5.8 MB/s | 18 MB 00:03 2024-09-13T15:22:37.845 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:22:37.845 INFO:teuthology.orchestra.run.smithi090.stdout:Total 14 MB/s | 53 MB 00:03 2024-09-13T15:22:37.916 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:22:37.953 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:22:37.953 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:22:38.595 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:22:38.595 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:22:38.607 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:22:38.697 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T15:22:38.724 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T15:22:38.774 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T15:22:39.138 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T15:22:39.190 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/32 2024-09-13T15:22:39.281 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/32 2024-09-13T15:22:39.365 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 6/32 2024-09-13T15:22:39.400 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/32 2024-09-13T15:22:39.463 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/32 2024-09-13T15:22:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T15:22:39.543 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x 9/32 2024-09-13T15:22:39.609 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/32 2024-09-13T15:22:39.635 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/32 2024-09-13T15:22:39.701 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/32 2024-09-13T15:22:39.807 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T15:22:39.853 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T15:22:39.908 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T15:22:39.940 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T15:22:39.952 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:22:39.967 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T15:22:40.004 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T15:22:40.037 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T15:22:40.056 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-13T15:22:40.334 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T15:22:40.335 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-13T15:22:40.382 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 20/32 2024-09-13T15:22:40.396 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-13T15:22:40.460 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 20/32 2024-09-13T15:22:40.487 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 21/32 2024-09-13T15:22:40.523 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T15:22:40.720 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-13T15:22:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/32 2024-09-13T15:22:40.926 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T15:22:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/32 2024-09-13T15:22:40.952 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 24/32 2024-09-13T15:22:41.023 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 24/32 2024-09-13T15:22:41.039 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 6/32 2024-09-13T15:22:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/32 2024-09-13T15:22:41.107 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 25/32 2024-09-13T15:22:41.127 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/32 2024-09-13T15:22:41.137 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86 26/32 2024-09-13T15:22:41.156 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-13T15:22:41.191 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x 9/32 2024-09-13T15:22:41.267 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/32 2024-09-13T15:22:41.301 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/32 2024-09-13T15:22:41.366 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/32 2024-09-13T15:22:41.473 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-13T15:22:41.508 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-13T15:22:41.569 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-13T15:22:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-13T15:22:41.634 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-13T15:22:41.680 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-13T15:22:41.713 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-13T15:22:42.013 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-13T15:22:42.068 INFO:teuthology.orchestra.run.smithi090.stdout: Upgrading : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 20/32 2024-09-13T15:22:42.158 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 20/32 2024-09-13T15:22:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 21/32 2024-09-13T15:22:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-13T15:22:42.638 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-13T15:22:42.662 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 24/32 2024-09-13T15:22:42.734 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 24/32 2024-09-13T15:22:42.810 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 25/32 2024-09-13T15:22:42.849 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86 26/32 2024-09-13T15:22:43.212 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/32 2024-09-13T15:22:43.258 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/32 2024-09-13T15:22:43.788 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/32 2024-09-13T15:22:43.836 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 28/32 2024-09-13T15:22:43.936 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 28/32 2024-09-13T15:22:43.936 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T15:22:43.936 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T15:22:43.936 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:43.971 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/32 2024-09-13T15:22:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/32 2024-09-13T15:22:45.006 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/32 2024-09-13T15:22:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 27/32 2024-09-13T15:22:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 28/32 2024-09-13T15:22:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 28/32 2024-09-13T15:22:45.691 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T15:22:45.691 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T15:22:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/32 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/32 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /root 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2024-09-13T15:22:53.900 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:54.634 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/32 2024-09-13T15:22:54.659 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/32 2024-09-13T15:22:54.660 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:22:54.660 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T15:22:54.660 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T15:22:54.660 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T15:22:54.660 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:54.661 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T15:22:54.736 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T15:22:54.736 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 6/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x 8/32 2024-09-13T15:22:55.911 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86 9/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 12/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T15:22:55.912 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T15:22:55.913 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 31/32 2024-09-13T15:22:56.467 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/32 2024-09-13T15:22:56.467 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /sys 2024-09-13T15:22:56.467 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /proc 2024-09-13T15:22:56.467 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /mnt 2024-09-13T15:22:56.468 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /var/tmp 2024-09-13T15:22:56.468 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /home 2024-09-13T15:22:56.468 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /root 2024-09-13T15:22:56.468 INFO:teuthology.orchestra.run.smithi090.stdout:skipping the directory /tmp 2024-09-13T15:22:56.468 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.961 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T15:22:56.962 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T15:22:56.963 INFO:teuthology.orchestra.run.smithi175.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T15:22:56.964 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:56.964 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:22:57.260 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/32 2024-09-13T15:22:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/32 2024-09-13T15:22:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:22:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T15:22:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T15:22:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T15:22:57.285 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:57.287 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T15:22:57.315 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-test 2024-09-13T15:22:57.379 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-13T15:22:57.379 INFO:teuthology.orchestra.run.smithi090.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T15:22:57.888 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:26 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:22:58.021 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:22:58.021 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:22:58.021 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 40 M 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 166 k 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2024-09-13T15:22:58.022 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:22:58.023 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 40 M 2024-09-13T15:22:58.023 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 164 M 2024-09-13T15:22:58.023 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:22:58.210 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-13T15:22:58.301 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 695 kB/s | 64 kB 00:00 2024-09-13T15:22:58.326 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): libcephsqlite-18.2.4-850.g50bbe7f3.el9.x 647 kB/s | 166 kB 00:00 2024-09-13T15:22:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T15:22:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/32 2024-09-13T15:22:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/32 2024-09-13T15:22:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/32 2024-09-13T15:22:58.710 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 6/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 7/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x 8/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86 9/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 10/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 11/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 12/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 13/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-13T15:22:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-13T15:22:58.712 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-13T15:22:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-13T15:22:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/32 2024-09-13T15:22:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-13T15:22:58.713 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 31/32 2024-09-13T15:22:59.369 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): ceph-test-18.2.4-850.g50bbe7f3.el9.x86_6 31 MB/s | 40 MB 00:01 2024-09-13T15:22:59.371 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:22:59.371 INFO:teuthology.orchestra.run.smithi175.stdout:Total 30 MB/s | 40 MB 00:01 2024-09-13T15:22:59.399 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:22:59.419 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:22:59.419 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:22:59.679 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout:Upgraded: 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-radosgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-selinux-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T15:22:59.680 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: libradosstriper1-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: librgw2-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-argparse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.681 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:22:59.682 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:22:59.747 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:22:59.747 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:00.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-test 2024-09-13T15:23:00.220 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:00.303 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T15:23:00.343 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T15:23:00.370 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/4 2024-09-13T15:23:00.640 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:29 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:00.777 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 40 M 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 166 k 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout:Install 4 Packages 2024-09-13T15:23:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 40 M 2024-09-13T15:23:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 164 M 2024-09-13T15:23:00.779 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:01.033 INFO:teuthology.orchestra.run.smithi090.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-13T15:23:01.059 INFO:teuthology.orchestra.run.smithi090.stdout:(2/4): libcephsqlite-18.2.4-850.g50bbe7f3.el9.x 713 kB/s | 166 kB 00:00 2024-09-13T15:23:01.084 INFO:teuthology.orchestra.run.smithi090.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-13T15:23:02.176 INFO:teuthology.orchestra.run.smithi090.stdout:(4/4): ceph-test-18.2.4-850.g50bbe7f3.el9.x86_6 29 MB/s | 40 MB 00:01 2024-09-13T15:23:02.178 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:02.179 INFO:teuthology.orchestra.run.smithi090.stdout:Total 29 MB/s | 40 MB 00:01 2024-09-13T15:23:02.206 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:02.226 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:02.226 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:02.555 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:02.555 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:03.143 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-13T15:23:03.209 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-13T15:23:03.237 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/4 2024-09-13T15:23:04.100 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/4 2024-09-13T15:23:04.176 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/4 2024-09-13T15:23:06.161 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/4 2024-09-13T15:23:06.161 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/4 2024-09-13T15:23:06.161 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/4 2024-09-13T15:23:06.161 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:06.490 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:06.803 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph 2024-09-13T15:23:07.027 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/4 2024-09-13T15:23:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/4 2024-09-13T15:23:07.380 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:36 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:07.515 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:07.517 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:07.517 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:07.517 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:07.517 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:07.517 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 6.5 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 2.1 M 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 1.5 M 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 246 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.7 M 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 17 M 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T15:23:07.518 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T15:23:07.519 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout:Install 33 Packages 2024-09-13T15:23:07.520 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:07.521 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 29 M 2024-09-13T15:23:07.521 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 104 M 2024-09-13T15:23:07.521 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:07.844 INFO:teuthology.orchestra.run.smithi175.stdout:(1/33): ceph-18.2.4-850.g50bbe7f3.el9.x86_64.rp 52 kB/s | 6.5 kB 00:00 2024-09-13T15:23:08.020 INFO:teuthology.orchestra.run.smithi175.stdout:(2/33): ceph-mgr-18.2.4-850.g50bbe7f3.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-13T15:23:08.070 INFO:teuthology.orchestra.run.smithi175.stdout:(3/33): ceph-mds-18.2.4-850.g50bbe7f3.el9.x86_6 6.0 MB/s | 2.1 MB 00:00 2024-09-13T15:23:08.178 INFO:teuthology.orchestra.run.smithi175.stdout:(4/33): ceph-mgr-modules-core-18.2.4-850.g50bbe 2.2 MB/s | 246 kB 00:00 2024-09-13T15:23:08.254 INFO:teuthology.orchestra.run.smithi175.stdout:(5/33): ceph-mon-18.2.4-850.g50bbe7f3.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-13T15:23:08.587 INFO:teuthology.orchestra.run.smithi175.stdout:(6/33): ceph-osd-18.2.4-850.g50bbe7f3.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-13T15:23:08.612 INFO:teuthology.orchestra.run.smithi175.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 584 kB/s | 253 kB 00:00 2024-09-13T15:23:08.671 INFO:teuthology.orchestra.run.smithi175.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-09-13T15:23:08.738 INFO:teuthology.orchestra.run.smithi175.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-09-13T15:23:08.763 INFO:teuthology.orchestra.run.smithi175.stdout:(10/33): python3-ply-3.11-14.el9.noarch.rpm 606 kB/s | 106 kB 00:00 2024-09-13T15:23:08.806 INFO:teuthology.orchestra.run.smithi175.stdout:(11/33): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-13T15:23:08.914 INFO:teuthology.orchestra.run.smithi175.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 321 kB/s | 35 kB 00:00 2024-09-13T15:23:08.981 INFO:teuthology.orchestra.run.smithi175.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 793 kB/s | 172 kB 00:00 2024-09-13T15:23:09.006 INFO:teuthology.orchestra.run.smithi175.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 553 kB/s | 51 kB 00:00 2024-09-13T15:23:09.040 INFO:teuthology.orchestra.run.smithi175.stdout:(15/33): python3-cryptography-36.0.1-4.el9.x86_ 1.6 MB/s | 1.2 MB 00:00 2024-09-13T15:23:09.065 INFO:teuthology.orchestra.run.smithi175.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 496 kB/s | 42 kB 00:00 2024-09-13T15:23:09.124 INFO:teuthology.orchestra.run.smithi175.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 371 kB/s | 43 kB 00:00 2024-09-13T15:23:09.157 INFO:teuthology.orchestra.run.smithi175.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 338 kB/s | 11 kB 00:00 2024-09-13T15:23:09.183 INFO:teuthology.orchestra.run.smithi175.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-13T15:23:09.199 INFO:teuthology.orchestra.run.smithi175.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-09-13T15:23:09.216 INFO:teuthology.orchestra.run.smithi175.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 695 kB/s | 23 kB 00:00 2024-09-13T15:23:09.233 INFO:teuthology.orchestra.run.smithi175.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 582 kB/s | 19 kB 00:00 2024-09-13T15:23:09.250 INFO:teuthology.orchestra.run.smithi175.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 582 kB/s | 20 kB 00:00 2024-09-13T15:23:09.267 INFO:teuthology.orchestra.run.smithi175.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-13T15:23:09.292 INFO:teuthology.orchestra.run.smithi175.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-13T15:23:09.318 INFO:teuthology.orchestra.run.smithi175.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-13T15:23:09.346 INFO:teuthology.orchestra.run.smithi175.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 308 kB/s | 16 kB 00:00 2024-09-13T15:23:09.371 INFO:teuthology.orchestra.run.smithi175.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.6 MB/s | 90 kB 00:00 2024-09-13T15:23:09.396 INFO:teuthology.orchestra.run.smithi175.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-13T15:23:09.422 INFO:teuthology.orchestra.run.smithi175.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-13T15:23:09.447 INFO:teuthology.orchestra.run.smithi175.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-13T15:23:09.472 INFO:teuthology.orchestra.run.smithi175.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-13T15:23:09.517 INFO:teuthology.orchestra.run.smithi175.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-13T15:23:09.518 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:09.518 INFO:teuthology.orchestra.run.smithi175.stdout:Total 15 MB/s | 29 MB 00:01 2024-09-13T15:23:09.631 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:09.647 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/4 2024-09-13T15:23:09.647 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/4 2024-09-13T15:23:09.647 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/4 2024-09-13T15:23:09.647 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-13T15:23:09.668 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:09.668 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-13T15:23:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-test-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout: libcephsqlite-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:09.962 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:10.071 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:10.072 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:10.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph 2024-09-13T15:23:10.944 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:39 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:11.089 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: ceph x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 6.5 k 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 2.1 M 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 1.5 M 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 246 k 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 4.7 M 2024-09-13T15:23:11.092 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 17 M 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T15:23:11.093 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T15:23:11.094 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:Install 33 Packages 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:11.095 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 29 M 2024-09-13T15:23:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 104 M 2024-09-13T15:23:11.096 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:11.293 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:11.367 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T15:23:11.400 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T15:23:11.432 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T15:23:11.532 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T15:23:11.627 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T15:23:11.668 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T15:23:11.711 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T15:23:11.741 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T15:23:11.776 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T15:23:11.809 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T15:23:11.850 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T15:23:11.884 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T15:23:11.928 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T15:23:11.979 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T15:23:12.003 INFO:teuthology.orchestra.run.smithi090.stdout:(1/33): ceph-18.2.4-850.g50bbe7f3.el9.x86_64.rp 36 kB/s | 6.5 kB 00:00 2024-09-13T15:23:12.056 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T15:23:12.193 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T15:23:12.195 INFO:teuthology.orchestra.run.smithi090.stdout:(2/33): ceph-mgr-18.2.4-850.g50bbe7f3.el9.x86_6 3.9 MB/s | 1.5 MB 00:00 2024-09-13T15:23:12.254 INFO:teuthology.orchestra.run.smithi090.stdout:(3/33): ceph-mds-18.2.4-850.g50bbe7f3.el9.x86_6 4.9 MB/s | 2.1 MB 00:00 2024-09-13T15:23:12.273 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T15:23:12.342 INFO:teuthology.orchestra.run.smithi090.stdout:(4/33): ceph-mgr-modules-core-18.2.4-850.g50bbe 2.8 MB/s | 246 kB 00:00 2024-09-13T15:23:12.351 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T15:23:12.390 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T15:23:12.439 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T15:23:12.442 INFO:teuthology.orchestra.run.smithi090.stdout:(5/33): ceph-mon-18.2.4-850.g50bbe7f3.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-13T15:23:12.503 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T15:23:12.696 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T15:23:12.752 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T15:23:12.776 INFO:teuthology.orchestra.run.smithi090.stdout:(6/33): ceph-osd-18.2.4-850.g50bbe7f3.el9.x86_6 28 MB/s | 17 MB 00:00 2024-09-13T15:23:12.822 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T15:23:12.926 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T15:23:12.968 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T15:23:13.085 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T15:23:13.249 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.n 28/33 2024-09-13T15:23:13.279 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/33 2024-09-13T15:23:13.307 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/33 2024-09-13T15:23:13.307 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:13.307 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T15:23:13.307 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T15:23:13.307 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T15:23:13.308 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:13.384 INFO:teuthology.orchestra.run.smithi090.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 243 kB/s | 253 kB 00:01 2024-09-13T15:23:13.501 INFO:teuthology.orchestra.run.smithi090.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 148 kB/s | 106 kB 00:00 2024-09-13T15:23:13.585 INFO:teuthology.orchestra.run.smithi090.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 1.1 MB/s | 1.2 MB 00:01 2024-09-13T15:23:14.077 INFO:teuthology.orchestra.run.smithi090.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 195 kB/s | 135 kB 00:00 2024-09-13T15:23:14.194 INFO:teuthology.orchestra.run.smithi090.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-13T15:23:14.229 INFO:teuthology.orchestra.run.smithi090.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 1.0 MB/s | 35 kB 00:00 2024-09-13T15:23:14.263 INFO:teuthology.orchestra.run.smithi090.stdout:(13/33): python3-requests-2.25.1-8.el9.noarch.r 164 kB/s | 125 kB 00:00 2024-09-13T15:23:14.288 INFO:teuthology.orchestra.run.smithi090.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 870 kB/s | 51 kB 00:00 2024-09-13T15:23:14.330 INFO:teuthology.orchestra.run.smithi090.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-09-13T15:23:14.355 INFO:teuthology.orchestra.run.smithi090.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-09-13T15:23:14.381 INFO:teuthology.orchestra.run.smithi090.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-09-13T15:23:14.423 INFO:teuthology.orchestra.run.smithi090.stdout:(18/33): python3-urllib3-1.26.5-6.el9.noarch.rp 256 kB/s | 215 kB 00:00 2024-09-13T15:23:14.448 INFO:teuthology.orchestra.run.smithi090.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 179 kB/s | 11 kB 00:00 2024-09-13T15:23:14.473 INFO:teuthology.orchestra.run.smithi090.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 365 kB/s | 18 kB 00:00 2024-09-13T15:23:14.507 INFO:teuthology.orchestra.run.smithi090.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 760 kB/s | 19 kB 00:00 2024-09-13T15:23:14.541 INFO:teuthology.orchestra.run.smithi090.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.9 MB/s | 358 kB 00:00 2024-09-13T15:23:14.566 INFO:teuthology.orchestra.run.smithi090.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 345 kB/s | 20 kB 00:00 2024-09-13T15:23:14.591 INFO:teuthology.orchestra.run.smithi090.stdout:(24/33): python3-jaraco-collections-3.0.0-8.el9 162 kB/s | 23 kB 00:00 2024-09-13T15:23:14.616 INFO:teuthology.orchestra.run.smithi090.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-09-13T15:23:14.642 INFO:teuthology.orchestra.run.smithi090.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-13T15:23:14.667 INFO:teuthology.orchestra.run.smithi090.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-13T15:23:14.701 INFO:teuthology.orchestra.run.smithi090.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-13T15:23:14.726 INFO:teuthology.orchestra.run.smithi090.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-09-13T15:23:14.760 INFO:teuthology.orchestra.run.smithi090.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-13T15:23:14.771 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/33 2024-09-13T15:23:14.785 INFO:teuthology.orchestra.run.smithi090.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-13T15:23:14.800 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/33 2024-09-13T15:23:14.800 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:14.800 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T15:23:14.800 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T15:23:14.800 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T15:23:14.801 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:14.811 INFO:teuthology.orchestra.run.smithi090.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-13T15:23:14.852 INFO:teuthology.orchestra.run.smithi090.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-09-13T15:23:14.859 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:14.859 INFO:teuthology.orchestra.run.smithi090.stdout:Total 7.8 MB/s | 29 MB 00:03 2024-09-13T15:23:14.979 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:15.016 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:15.016 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:15.247 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 31/33 2024-09-13T15:23:15.272 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 31/33 2024-09-13T15:23:15.272 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:15.272 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T15:23:15.272 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T15:23:15.272 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T15:23:15.273 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:15.416 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:15.417 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:15.481 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 32/33 2024-09-13T15:23:15.505 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 32/33 2024-09-13T15:23:15.505 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:15.505 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T15:23:15.505 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T15:23:15.505 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T15:23:15.505 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:15.570 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 33/33 2024-09-13T15:23:16.687 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:16.757 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-13T15:23:16.790 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-13T15:23:16.839 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-13T15:23:16.932 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-13T15:23:17.010 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-13T15:23:17.059 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-13T15:23:17.103 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-13T15:23:17.133 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-13T15:23:17.167 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-13T15:23:17.201 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-13T15:23:17.241 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-13T15:23:17.286 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-13T15:23:17.321 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-13T15:23:17.370 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 33/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.n 6/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T15:23:17.426 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T15:23:17.427 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T15:23:17.428 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T15:23:17.428 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T15:23:17.428 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T15:23:17.428 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T15:23:17.429 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T15:23:17.457 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-13T15:23:17.613 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-13T15:23:17.694 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-13T15:23:17.747 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-13T15:23:17.787 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-13T15:23:17.844 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-13T15:23:17.908 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-13T15:23:18.101 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-13T15:23:18.159 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-13T15:23:18.237 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-13T15:23:18.317 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T15:23:18.317 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:18.317 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T15:23:18.318 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T15:23:18.319 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:18.320 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:18.327 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-13T15:23:18.368 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-13T15:23:18.481 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-13T15:23:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.n 28/33 2024-09-13T15:23:18.687 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/33 2024-09-13T15:23:18.696 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-base 2024-09-13T15:23:18.713 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 29/33 2024-09-13T15:23:18.713 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:18.713 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T15:23:18.713 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T15:23:18.713 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T15:23:18.713 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:19.265 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:48 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:19.354 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:23:19.396 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:19.397 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:23:19.397 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:19.467 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install cephadm 2024-09-13T15:23:20.027 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:49 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:20.157 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 226 k 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:23:20.158 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:20.159 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 226 k 2024-09-13T15:23:20.159 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 217 k 2024-09-13T15:23:20.159 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:20.215 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/33 2024-09-13T15:23:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 30/33 2024-09-13T15:23:20.244 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:20.244 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T15:23:20.244 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T15:23:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T15:23:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:20.388 INFO:teuthology.orchestra.run.smithi175.stdout:cephadm-18.2.4-850.g50bbe7f3.el9.noarch.rpm 986 kB/s | 226 kB 00:00 2024-09-13T15:23:20.389 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:20.389 INFO:teuthology.orchestra.run.smithi175.stdout:Total 979 kB/s | 226 kB 00:00 2024-09-13T15:23:20.389 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:20.394 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:20.394 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:20.397 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:20.397 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:20.471 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:20.700 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 31/33 2024-09-13T15:23:20.725 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 31/33 2024-09-13T15:23:20.725 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:20.725 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T15:23:20.725 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T15:23:20.725 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T15:23:20.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:20.910 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:20.926 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 32/33 2024-09-13T15:23:20.951 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 32/33 2024-09-13T15:23:20.951 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:20.951 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T15:23:20.951 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T15:23:20.951 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T15:23:20.951 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:21.009 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:21.014 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 33/33 2024-09-13T15:23:21.413 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:21.690 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:21.690 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:21.690 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:21.690 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:21.690 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:21.690 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:21.918 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T15:23:22.486 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:51 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:22.620 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:22.621 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:22.621 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-13T15:23:22.621 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:22.621 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 143 k 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 143 k 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 395 k 2024-09-13T15:23:22.622 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:22.819 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-immutable-object-cache-18.2.4-850.g50bbe7f 725 kB/s | 143 kB 00:00 2024-09-13T15:23:22.820 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:22.820 INFO:teuthology.orchestra.run.smithi175.stdout:Total 718 kB/s | 143 kB 00:00 2024-09-13T15:23:22.820 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:22.828 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:22.828 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:22.885 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:22.886 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 33/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 3/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 4/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 5/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.n 6/33 2024-09-13T15:23:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-13T15:23:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-13T15:23:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-13T15:23:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-13T15:23:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-13T15:23:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-13T15:23:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-13T15:23:23.051 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:23.136 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.e 1/1 2024-09-13T15:23:23.161 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.e 1/1 2024-09-13T15:23:23.161 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:23.161 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T15:23:23.161 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:23.923 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-13T15:23:23.923 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:23.923 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:23.923 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mds-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mon-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-osd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T15:23:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T15:23:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:23.926 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:24.095 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.e 1/1 2024-09-13T15:23:24.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:24.095 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:24.095 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:24.095 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:24.095 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:24.349 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr 2024-09-13T15:23:24.389 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-base 2024-09-13T15:23:24.924 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:24.956 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:53 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:25.019 INFO:teuthology.orchestra.run.smithi175.stdout:Package ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:23:25.047 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-base-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:23:25.063 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:25.064 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:23:25.064 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:25.088 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:25.089 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:23:25.089 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:25.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T15:23:25.168 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install cephadm 2024-09-13T15:23:25.693 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:25.726 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:54 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:25.824 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 3.5 M 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 24 k 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 16 k 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T15:23:25.825 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout:Install 5 Packages 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.8 M 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 87 M 2024-09-13T15:23:25.826 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:25.856 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:25.857 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:25.857 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 226 k 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 226 k 2024-09-13T15:23:25.858 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 217 k 2024-09-13T15:23:25.859 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:26.039 INFO:teuthology.orchestra.run.smithi175.stdout:(1/5): ceph-prometheus-alerts-18.2.4-850.g50bbe 105 kB/s | 16 kB 00:00 2024-09-13T15:23:26.067 INFO:teuthology.orchestra.run.smithi175.stdout:(2/5): ceph-grafana-dashboards-18.2.4-850.g50bb 132 kB/s | 24 kB 00:00 2024-09-13T15:23:26.106 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm-18.2.4-850.g50bbe7f3.el9.noarch.rpm 911 kB/s | 226 kB 00:00 2024-09-13T15:23:26.107 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:26.107 INFO:teuthology.orchestra.run.smithi090.stdout:Total 906 kB/s | 226 kB 00:00 2024-09-13T15:23:26.107 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:26.112 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:26.112 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:26.115 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:26.115 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:26.125 INFO:teuthology.orchestra.run.smithi175.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 359 kB/s | 31 kB 00:00 2024-09-13T15:23:26.159 INFO:teuthology.orchestra.run.smithi175.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-09-13T15:23:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:26.259 INFO:teuthology.orchestra.run.smithi175.stdout:(5/5): ceph-mgr-dashboard-18.2.4-850.g50bbe7f3. 9.4 MB/s | 3.5 MB 00:00 2024-09-13T15:23:26.264 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:26.265 INFO:teuthology.orchestra.run.smithi175.stdout:Total 8.7 MB/s | 3.8 MB 00:00 2024-09-13T15:23:26.265 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:26.270 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:26.270 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:26.351 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:26.351 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:26.603 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:26.682 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T15:23:26.722 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T15:23:26.746 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:26.759 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.no 3/5 2024-09-13T15:23:27.170 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:27.490 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/1 2024-09-13T15:23:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:27.491 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:27.491 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:27.491 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:27.772 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-immutable-object-cache 2024-09-13T15:23:27.812 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.n 4/5 2024-09-13T15:23:27.854 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 5/5 2024-09-13T15:23:28.344 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:57 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:28.478 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:28.478 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:28.478 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-13T15:23:28.478 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:28.478 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 143 k 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 143 k 2024-09-13T15:23:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 395 k 2024-09-13T15:23:28.480 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:28.660 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-immutable-object-cache-18.2.4-850.g50bbe7f 790 kB/s | 143 kB 00:00 2024-09-13T15:23:28.661 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:28.661 INFO:teuthology.orchestra.run.smithi090.stdout:Total 784 kB/s | 143 kB 00:00 2024-09-13T15:23:28.661 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:28.669 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:28.669 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:28.727 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:28.727 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:28.884 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:28.978 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 5/5 2024-09-13T15:23:28.978 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.n 1/5 2024-09-13T15:23:28.978 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 2/5 2024-09-13T15:23:28.979 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.no 3/5 2024-09-13T15:23:28.979 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T15:23:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.e 1/1 2024-09-13T15:23:29.012 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.e 1/1 2024-09-13T15:23:29.012 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:29.012 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T15:23:29.012 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:29.251 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T15:23:29.251 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:29.251 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:29.251 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:29.251 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:29.251 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:29.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T15:23:29.252 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T15:23:29.252 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:29.252 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:29.486 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T15:23:29.995 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.e 1/1 2024-09-13T15:23:29.995 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:29.995 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:29.995 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-immutable-object-cache-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:29.995 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:29.995 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:30.051 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:30.185 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:30.186 INFO:teuthology.orchestra.run.smithi175.stdout:====================================================================================== 2024-09-13T15:23:30.186 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:30.186 INFO:teuthology.orchestra.run.smithi175.stdout:====================================================================================== 2024-09-13T15:23:30.186 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 7.4 M 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T15:23:30.187 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout:====================================================================================== 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout:Install 12 Packages 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 38 M 2024-09-13T15:23:30.188 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 212 M 2024-09-13T15:23:30.189 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:30.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr 2024-09-13T15:23:30.658 INFO:teuthology.orchestra.run.smithi175.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-13T15:23:30.725 INFO:teuthology.orchestra.run.smithi175.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-09-13T15:23:30.767 INFO:teuthology.orchestra.run.smithi175.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-13T15:23:30.851 INFO:teuthology.orchestra.run.smithi175.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 218 kB/s | 18 kB 00:00 2024-09-13T15:23:30.868 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:00:59 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:30.942 INFO:teuthology.orchestra.run.smithi175.stdout:(5/12): ceph-mgr-diskprediction-local-18.2.4-85 16 MB/s | 7.4 MB 00:00 2024-09-13T15:23:30.961 INFO:teuthology.orchestra.run.smithi090.stdout:Package ceph-mgr-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:23:30.967 INFO:teuthology.orchestra.run.smithi175.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 330 kB/s | 38 kB 00:00 2024-09-13T15:23:31.002 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:31.003 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:23:31.003 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:31.026 INFO:teuthology.orchestra.run.smithi175.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-09-13T15:23:31.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-dashboard 2024-09-13T15:23:31.085 INFO:teuthology.orchestra.run.smithi175.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.0 MB/s | 245 kB 00:00 2024-09-13T15:23:31.177 INFO:teuthology.orchestra.run.smithi175.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-09-13T15:23:31.252 INFO:teuthology.orchestra.run.smithi175.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-13T15:23:31.377 INFO:teuthology.orchestra.run.smithi175.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 11 MB/s | 4.9 MB 00:00 2024-09-13T15:23:31.635 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:00 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:31.677 INFO:teuthology.orchestra.run.smithi175.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-13T15:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:31.679 INFO:teuthology.orchestra.run.smithi175.stdout:Total 26 MB/s | 38 MB 00:01 2024-09-13T15:23:31.765 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 3.5 M 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 24 k 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 16 k 2024-09-13T15:23:31.766 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout:Install 5 Packages 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:31.767 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.8 M 2024-09-13T15:23:31.768 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 87 M 2024-09-13T15:23:31.768 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:32.035 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:32.062 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:32.062 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:32.378 INFO:teuthology.orchestra.run.smithi090.stdout:(1/5): ceph-grafana-dashboards-18.2.4-850.g50bb 152 kB/s | 24 kB 00:00 2024-09-13T15:23:32.395 INFO:teuthology.orchestra.run.smithi090.stdout:(2/5): ceph-prometheus-alerts-18.2.4-850.g50bbe 93 kB/s | 16 kB 00:00 2024-09-13T15:23:32.437 INFO:teuthology.orchestra.run.smithi090.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 525 kB/s | 31 kB 00:00 2024-09-13T15:23:32.480 INFO:teuthology.orchestra.run.smithi090.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-13T15:23:32.556 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:32.556 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:32.605 INFO:teuthology.orchestra.run.smithi090.stdout:(5/5): ceph-mgr-dashboard-18.2.4-850.g50bbe7f3. 9.1 MB/s | 3.5 MB 00:00 2024-09-13T15:23:32.608 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:32.608 INFO:teuthology.orchestra.run.smithi090.stdout:Total 4.5 MB/s | 3.8 MB 00:00 2024-09-13T15:23:32.608 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:32.615 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:32.616 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:32.704 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:32.705 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:33.052 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-13T15:23:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-13T15:23:33.130 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.no 3/5 2024-09-13T15:23:33.311 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:33.405 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T15:23:33.450 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T15:23:33.551 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T15:23:33.601 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T15:23:33.786 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T15:23:33.817 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T15:23:33.893 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T15:23:34.174 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.n 4/5 2024-09-13T15:23:34.261 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 5/5 2024-09-13T15:23:34.575 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T15:23:34.705 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T15:23:35.837 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 5/5 2024-09-13T15:23:35.837 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.n 1/5 2024-09-13T15:23:35.838 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 2/5 2024-09-13T15:23:35.838 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.no 3/5 2024-09-13T15:23:35.838 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-13T15:23:35.949 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-grafana-dashboards-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-prometheus-alerts-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:36.088 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T15:23:36.089 INFO:teuthology.orchestra.run.smithi090.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T15:23:36.089 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:36.089 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:36.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-13T15:23:36.904 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:05 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:36.980 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T15:23:37.016 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7 12/12 2024-09-13T15:23:37.035 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 7.4 M 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T15:23:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout:====================================================================================== 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout:Install 12 Packages 2024-09-13T15:23:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:37.038 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 38 M 2024-09-13T15:23:37.038 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 212 M 2024-09-13T15:23:37.038 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:37.565 INFO:teuthology.orchestra.run.smithi090.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 765 kB/s | 191 kB 00:00 2024-09-13T15:23:37.666 INFO:teuthology.orchestra.run.smithi090.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-09-13T15:23:37.691 INFO:teuthology.orchestra.run.smithi090.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 260 kB/s | 33 kB 00:00 2024-09-13T15:23:37.816 INFO:teuthology.orchestra.run.smithi090.stdout:(4/12): ceph-mgr-diskprediction-local-18.2.4-85 15 MB/s | 7.4 MB 00:00 2024-09-13T15:23:37.833 INFO:teuthology.orchestra.run.smithi090.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 128 kB/s | 18 kB 00:00 2024-09-13T15:23:37.926 INFO:teuthology.orchestra.run.smithi090.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 350 kB/s | 38 kB 00:00 2024-09-13T15:23:38.027 INFO:teuthology.orchestra.run.smithi090.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 26 MB/s | 4.9 MB 00:00 2024-09-13T15:23:38.061 INFO:teuthology.orchestra.run.smithi090.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-13T15:23:38.144 INFO:teuthology.orchestra.run.smithi090.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-13T15:23:38.245 INFO:teuthology.orchestra.run.smithi090.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-13T15:23:38.553 INFO:teuthology.orchestra.run.smithi090.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-09-13T15:23:38.612 INFO:teuthology.orchestra.run.smithi090.stdout:(12/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rp 3.2 MB/s | 3.0 MB 00:00 2024-09-13T15:23:38.613 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:38.613 INFO:teuthology.orchestra.run.smithi090.stdout:Total 24 MB/s | 38 MB 00:01 2024-09-13T15:23:38.968 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:38.995 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:38.995 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:39.498 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7 12/12 2024-09-13T15:23:39.498 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7 1/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T15:23:39.499 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T15:23:39.522 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:39.523 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T15:23:39.893 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:39.893 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T15:23:39.894 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T15:23:39.895 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:39.895 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:40.258 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-rook 2024-09-13T15:23:40.280 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:40.354 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-13T15:23:40.402 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-13T15:23:40.501 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-13T15:23:40.545 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-13T15:23:40.731 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-13T15:23:40.761 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-13T15:23:40.831 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:09 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-13T15:23:40.964 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:40.965 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:40.965 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:40.965 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:40.965 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 49 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T15:23:40.966 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout:Install 13 Packages 2024-09-13T15:23:40.967 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:40.968 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 2.2 M 2024-09-13T15:23:40.968 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 26 M 2024-09-13T15:23:40.968 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:41.340 INFO:teuthology.orchestra.run.smithi175.stdout:(1/13): ceph-mgr-rook-18.2.4-850.g50bbe7f3.el9. 292 kB/s | 49 kB 00:00 2024-09-13T15:23:41.506 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-13T15:23:41.619 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-13T15:23:41.690 INFO:teuthology.orchestra.run.smithi175.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-13T15:23:41.732 INFO:teuthology.orchestra.run.smithi175.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-13T15:23:42.166 INFO:teuthology.orchestra.run.smithi175.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 269 kB/s | 222 kB 00:00 2024-09-13T15:23:42.433 INFO:teuthology.orchestra.run.smithi175.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 214 kB/s | 159 kB 00:00 2024-09-13T15:23:42.483 INFO:teuthology.orchestra.run.smithi175.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 649 kB/s | 32 kB 00:00 2024-09-13T15:23:42.508 INFO:teuthology.orchestra.run.smithi175.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 359 kB/s | 279 kB 00:00 2024-09-13T15:23:42.534 INFO:teuthology.orchestra.run.smithi175.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 284 kB/s | 14 kB 00:00 2024-09-13T15:23:42.592 INFO:teuthology.orchestra.run.smithi175.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.6 MB/s | 220 kB 00:00 2024-09-13T15:23:42.626 INFO:teuthology.orchestra.run.smithi175.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-13T15:23:42.659 INFO:teuthology.orchestra.run.smithi175.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-13T15:23:42.701 INFO:teuthology.orchestra.run.smithi175.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-13T15:23:42.827 INFO:teuthology.orchestra.run.smithi175.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 81 kB/s | 54 kB 00:00 2024-09-13T15:23:42.830 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:42.831 INFO:teuthology.orchestra.run.smithi175.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-13T15:23:42.894 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T15:23:42.900 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:42.916 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:42.917 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:43.007 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:43.008 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:43.539 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:43.724 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T15:23:43.785 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T15:23:43.849 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T15:23:43.887 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T15:23:43.928 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T15:23:43.943 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-13T15:23:43.977 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7 12/12 2024-09-13T15:23:44.029 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T15:23:44.139 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T15:23:44.205 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T15:23:44.916 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T15:23:44.977 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T15:23:45.019 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T15:23:45.075 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T15:23:45.097 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 13/13 2024-09-13T15:23:45.907 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 13/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 1/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T15:23:45.908 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T15:23:46.258 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:46.259 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:46.510 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T15:23:47.084 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:16 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:47.216 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 139 k 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T15:23:47.217 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout:Install 6 Packages 2024-09-13T15:23:47.218 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:47.219 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 7.0 M 2024-09-13T15:23:47.219 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 33 M 2024-09-13T15:23:47.219 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7 12/12 2024-09-13T15:23:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7 1/12 2024-09-13T15:23:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-13T15:23:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-diskprediction-local-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T15:23:47.694 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:47.695 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:47.945 INFO:teuthology.orchestra.run.smithi175.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-13T15:23:48.037 INFO:teuthology.orchestra.run.smithi175.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 5.8 MB/s | 548 kB 00:00 2024-09-13T15:23:48.062 INFO:teuthology.orchestra.run.smithi175.stdout:(3/6): ceph-mgr-cephadm-18.2.4-850.g50bbe7f3.el 538 kB/s | 139 kB 00:00 2024-09-13T15:23:48.088 INFO:teuthology.orchestra.run.smithi175.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T15:23:48.119 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-rook 2024-09-13T15:23:48.171 INFO:teuthology.orchestra.run.smithi175.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-13T15:23:48.205 INFO:teuthology.orchestra.run.smithi175.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 524 kB/s | 74 kB 00:00 2024-09-13T15:23:48.207 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:48.208 INFO:teuthology.orchestra.run.smithi175.stdout:Total 7.1 MB/s | 7.0 MB 00:00 2024-09-13T15:23:48.279 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:48.290 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:48.291 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:48.389 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:48.390 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:48.688 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:17 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:48.694 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:48.820 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:48.821 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:48.821 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:48.821 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 49 k 2024-09-13T15:23:48.821 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:48.822 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:48.823 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:48.823 INFO:teuthology.orchestra.run.smithi090.stdout:Install 13 Packages 2024-09-13T15:23:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:48.823 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.2 M 2024-09-13T15:23:48.823 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 26 M 2024-09-13T15:23:48.823 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:48.843 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T15:23:48.890 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T15:23:49.335 INFO:teuthology.orchestra.run.smithi090.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 251 kB/s | 19 kB 00:00 2024-09-13T15:23:49.360 INFO:teuthology.orchestra.run.smithi090.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 252 kB/s | 26 kB 00:00 2024-09-13T15:23:49.417 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T15:23:49.419 INFO:teuthology.orchestra.run.smithi090.stdout:(3/13): ceph-mgr-rook-18.2.4-850.g50bbe7f3.el9. 307 kB/s | 49 kB 00:00 2024-09-13T15:23:49.484 INFO:teuthology.orchestra.run.smithi090.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-13T15:23:49.512 INFO:teuthology.orchestra.run.smithi090.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-09-13T15:23:49.517 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T15:23:49.537 INFO:teuthology.orchestra.run.smithi090.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-13T15:23:49.579 INFO:teuthology.orchestra.run.smithi090.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-13T15:23:49.588 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T15:23:49.604 INFO:teuthology.orchestra.run.smithi090.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 210 kB/s | 14 kB 00:00 2024-09-13T15:23:49.619 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 6/6 2024-09-13T15:23:49.646 INFO:teuthology.orchestra.run.smithi090.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-13T15:23:49.672 INFO:teuthology.orchestra.run.smithi090.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.4 MB/s | 220 kB 00:00 2024-09-13T15:23:49.714 INFO:teuthology.orchestra.run.smithi090.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.5 MB/s | 90 kB 00:00 2024-09-13T15:23:49.739 INFO:teuthology.orchestra.run.smithi090.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-09-13T15:23:49.781 INFO:teuthology.orchestra.run.smithi090.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-13T15:23:49.784 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:49.784 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-13T15:23:49.850 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:49.866 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:49.866 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:49.957 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:49.957 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:50.417 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 6/6 2024-09-13T15:23:50.417 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/6 2024-09-13T15:23:50.417 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T15:23:50.417 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T15:23:50.418 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T15:23:50.418 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T15:23:50.462 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:50.626 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T15:23:50.689 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T15:23:50.690 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T15:23:50.690 INFO:teuthology.orchestra.run.smithi175.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T15:23:50.690 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:50.690 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:50.692 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-13T15:23:50.756 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-13T15:23:50.793 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-13T15:23:50.835 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-13T15:23:50.946 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-13T15:23:50.957 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-fuse 2024-09-13T15:23:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-13T15:23:51.113 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-13T15:23:51.518 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:20 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:51.649 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 852 k 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:51.650 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 932 k 2024-09-13T15:23:51.651 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 2.7 M 2024-09-13T15:23:51.652 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:51.817 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-13T15:23:51.879 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-13T15:23:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-13T15:23:51.923 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 536 kB/s | 80 kB 00:00 2024-09-13T15:23:51.961 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-13T15:23:51.982 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 13/13 2024-09-13T15:23:52.048 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-fuse-18.2.4-850.g50bbe7f3.el9.x86_6 3.0 MB/s | 852 kB 00:00 2024-09-13T15:23:52.049 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:52.049 INFO:teuthology.orchestra.run.smithi175.stdout:Total 2.3 MB/s | 932 kB 00:00 2024-09-13T15:23:52.060 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:52.082 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:52.082 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:52.165 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:52.165 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:52.356 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:52.496 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T15:23:52.571 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/2 2024-09-13T15:23:52.932 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 13/13 2024-09-13T15:23:52.932 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 1/13 2024-09-13T15:23:52.932 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-13T15:23:52.933 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-13T15:23:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-13T15:23:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:53.264 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-rook-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T15:23:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:53.266 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:53.293 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/2 2024-09-13T15:23:53.293 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/2 2024-09-13T15:23:53.571 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T15:23:53.572 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:53.572 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:53.572 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T15:23:53.572 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:53.572 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:53.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-mgr-cephadm 2024-09-13T15:23:53.852 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-volume 2024-09-13T15:23:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:23 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:54.295 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 139 k 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T15:23:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:54.297 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 7.0 M 2024-09-13T15:23:54.298 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 33 M 2024-09-13T15:23:54.298 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:54.436 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:23 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:54.572 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:54.572 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:54.572 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-09-13T15:23:54.572 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 264 k 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout:Install 2 Packages 2024-09-13T15:23:54.573 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:54.574 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 342 k 2024-09-13T15:23:54.574 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 1.5 M 2024-09-13T15:23:54.574 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:54.683 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-13T15:23:54.709 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): ceph-mgr-cephadm-18.2.4-850.g50bbe7f3.el 561 kB/s | 139 kB 00:00 2024-09-13T15:23:54.759 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T15:23:54.801 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 1.7 MB/s | 74 kB 00:00 2024-09-13T15:23:54.893 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-13T15:23:54.926 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-09-13T15:23:54.929 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:54.929 INFO:teuthology.orchestra.run.smithi090.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-13T15:23:54.986 INFO:teuthology.orchestra.run.smithi175.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 498 kB/s | 77 kB 00:00 2024-09-13T15:23:54.999 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:55.011 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:55.011 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:55.079 INFO:teuthology.orchestra.run.smithi175.stdout:(2/2): ceph-volume-18.2.4-850.g50bbe7f3.el9.noa 1.0 MB/s | 264 kB 00:00 2024-09-13T15:23:55.079 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:55.079 INFO:teuthology.orchestra.run.smithi175.stdout:Total 675 kB/s | 342 kB 00:00 2024-09-13T15:23:55.091 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:55.101 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:55.101 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:55.110 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:55.110 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:55.139 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:55.139 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:55.280 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:55.425 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:55.455 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-13T15:23:55.489 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2/2 2024-09-13T15:23:55.505 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2/2 2024-09-13T15:23:55.505 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:23:55.505 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T15:23:55.505 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:55.601 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-13T15:23:55.655 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-13T15:23:56.188 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-13T15:23:56.212 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 1/2 2024-09-13T15:23:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-13T15:23:56.372 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-13T15:23:56.402 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 6/6 2024-09-13T15:23:56.523 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-13T15:23:56.523 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:56.524 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:56.524 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:56.524 INFO:teuthology.orchestra.run.smithi175.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T15:23:56.524 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:56.524 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:56.845 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados-devel 2024-09-13T15:23:57.426 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:26 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:57.523 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 6/6 2024-09-13T15:23:57.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 1/6 2024-09-13T15:23:57.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-13T15:23:57.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-13T15:23:57.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-13T15:23:57.524 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-13T15:23:57.560 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:57.561 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:57.561 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-13T15:23:57.561 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:57.561 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 127 k 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 127 k 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 456 k 2024-09-13T15:23:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:23:57.759 INFO:teuthology.orchestra.run.smithi175.stdout:librados-devel-18.2.4-850.g50bbe7f3.el9.x86_64. 643 kB/s | 127 kB 00:00 2024-09-13T15:23:57.760 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:57.760 INFO:teuthology.orchestra.run.smithi175.stdout:Total 638 kB/s | 127 kB 00:00 2024-09-13T15:23:57.760 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:23:57.766 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:23:57.766 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:23:57.790 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-13T15:23:57.790 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:57.790 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:23:57.790 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-cephadm-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:23:57.790 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T15:23:57.790 INFO:teuthology.orchestra.run.smithi090.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T15:23:57.791 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T15:23:57.791 INFO:teuthology.orchestra.run.smithi090.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T15:23:57.791 INFO:teuthology.orchestra.run.smithi090.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T15:23:57.791 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:57.791 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:23:57.834 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:23:57.834 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:23:58.005 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:23:58.103 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-fuse 2024-09-13T15:23:58.114 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:23:58.560 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:23:58.668 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:27 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:58.800 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 852 k 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:23:58.801 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-09-13T15:23:58.802 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:23:58.802 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 932 k 2024-09-13T15:23:58.802 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 2.7 M 2024-09-13T15:23:58.802 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:23:58.855 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:23:58.855 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:58.855 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:23:58.855 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:23:58.855 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:23:58.855 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:59.009 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 496 kB/s | 80 kB 00:00 2024-09-13T15:23:59.101 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs2 2024-09-13T15:23:59.143 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): ceph-fuse-18.2.4-850.g50bbe7f3.el9.x86_6 2.8 MB/s | 852 kB 00:00 2024-09-13T15:23:59.143 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:23:59.143 INFO:teuthology.orchestra.run.smithi090.stdout:Total 2.7 MB/s | 932 kB 00:00 2024-09-13T15:23:59.155 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:23:59.176 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:23:59.176 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:23:59.259 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:23:59.260 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:23:59.446 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:23:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-13T15:23:59.659 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/2 2024-09-13T15:23:59.688 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:28 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:23:59.783 INFO:teuthology.orchestra.run.smithi175.stdout:Package libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:23:59.828 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:23:59.829 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:23:59.829 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:23:59.904 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install libcephfs-devel 2024-09-13T15:24:00.350 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2/2 2024-09-13T15:24:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/2 2024-09-13T15:24:00.470 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:29 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:00.604 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 31 k 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 31 k 2024-09-13T15:24:00.605 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 139 k 2024-09-13T15:24:00.606 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:24:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-13T15:24:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:00.630 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-13T15:24:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:00.630 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:00.745 INFO:teuthology.orchestra.run.smithi175.stdout:libcephfs-devel-18.2.4-850.g50bbe7f3.el9.x86_64 220 kB/s | 31 kB 00:00 2024-09-13T15:24:00.746 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:00.746 INFO:teuthology.orchestra.run.smithi175.stdout:Total 218 kB/s | 31 kB 00:00 2024-09-13T15:24:00.746 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:24:00.749 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:24:00.750 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:24:00.773 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:24:00.774 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:24:00.866 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:24:00.975 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:00.979 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install ceph-volume 2024-09-13T15:24:01.363 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:01.550 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:30 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:01.605 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:01.605 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:01.605 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:24:01.605 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:01.605 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:01.605 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:01.687 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repository Size 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume noarch 2:18.2.4-850.g50bbe7f3.el9 ceph-noarch 264 k 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T15:24:01.688 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:01.689 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:24:01.689 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:01.689 INFO:teuthology.orchestra.run.smithi090.stdout:Install 2 Packages 2024-09-13T15:24:01.689 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:01.689 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 342 k 2024-09-13T15:24:01.690 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 1.5 M 2024-09-13T15:24:01.690 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:24:01.787 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librados2 2024-09-13T15:24:01.945 INFO:teuthology.orchestra.run.smithi090.stdout:(1/2): ceph-volume-18.2.4-850.g50bbe7f3.el9.noa 1.1 MB/s | 264 kB 00:00 2024-09-13T15:24:02.087 INFO:teuthology.orchestra.run.smithi090.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 205 kB/s | 77 kB 00:00 2024-09-13T15:24:02.087 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:02.088 INFO:teuthology.orchestra.run.smithi090.stdout:Total 855 kB/s | 342 kB 00:00 2024-09-13T15:24:02.100 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:24:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:24:02.110 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:24:02.150 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:24:02.150 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:24:02.296 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:24:02.349 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:31 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:02.440 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:02.483 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:02.484 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:24:02.484 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:02.497 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-13T15:24:02.531 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2/2 2024-09-13T15:24:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2/2 2024-09-13T15:24:02.548 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:24:02.548 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T15:24:02.548 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:02.565 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install librbd1 2024-09-13T15:24:03.133 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:03.223 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:03.266 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:03.267 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:24:03.267 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:03.309 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 1/2 2024-09-13T15:24:03.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rados 2024-09-13T15:24:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-13T15:24:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:03.626 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-volume-2:18.2.4-850.g50bbe7f3.el9.noarch 2024-09-13T15:24:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T15:24:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:03.626 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:03.904 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:32 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:03.994 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:04.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados-devel 2024-09-13T15:24:04.037 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:04.038 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:24:04.038 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:04.118 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rgw 2024-09-13T15:24:04.595 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:04.673 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:33 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:04.730 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:04.730 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:04.730 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 127 k 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:24:04.731 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:04.732 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 127 k 2024-09-13T15:24:04.732 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 456 k 2024-09-13T15:24:04.732 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:24:04.763 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:04.805 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:04.806 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:24:04.806 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:04.873 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-cephfs 2024-09-13T15:24:04.923 INFO:teuthology.orchestra.run.smithi090.stdout:librados-devel-18.2.4-850.g50bbe7f3.el9.x86_64. 662 kB/s | 127 kB 00:00 2024-09-13T15:24:04.923 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:04.924 INFO:teuthology.orchestra.run.smithi090.stdout:Total 658 kB/s | 127 kB 00:00 2024-09-13T15:24:04.924 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:24:04.929 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:24:04.929 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:24:04.998 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:24:04.999 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:24:05.153 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:24:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:05.438 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:34 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:05.529 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:05.571 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:05.572 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:24:05.573 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:05.648 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install python3-rbd 2024-09-13T15:24:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:06.032 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: librados-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:06.032 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:06.217 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:06.308 INFO:teuthology.orchestra.run.smithi175.stdout:Package python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:06.343 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs2 2024-09-13T15:24:06.350 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:06.351 INFO:teuthology.orchestra.run.smithi175.stdout:Nothing to do. 2024-09-13T15:24:06.351 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:06.423 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-fuse 2024-09-13T15:24:06.921 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:06.996 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:35 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:07.013 INFO:teuthology.orchestra.run.smithi090.stdout:Package libcephfs2-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:07.058 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:07.059 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:07.059 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:07.129 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 87 k 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:07.130 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 87 k 2024-09-13T15:24:07.131 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 227 k 2024-09-13T15:24:07.131 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:24:07.137 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install libcephfs-devel 2024-09-13T15:24:07.323 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-fuse-18.2.4-850.g50bbe7f3.el9.x86_64.rpm 450 kB/s | 87 kB 00:00 2024-09-13T15:24:07.324 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:07.324 INFO:teuthology.orchestra.run.smithi175.stdout:Total 447 kB/s | 87 kB 00:00 2024-09-13T15:24:07.324 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:24:07.331 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:24:07.331 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:24:07.394 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:24:07.394 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:24:07.541 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:24:07.636 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:07.698 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:36 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:07.831 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 31 k 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:07.832 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:24:07.833 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:07.833 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 31 k 2024-09-13T15:24:07.833 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 139 k 2024-09-13T15:24:07.833 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:24:08.023 INFO:teuthology.orchestra.run.smithi090.stdout:libcephfs-devel-18.2.4-850.g50bbe7f3.el9.x86_64 163 kB/s | 31 kB 00:00 2024-09-13T15:24:08.024 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:08.024 INFO:teuthology.orchestra.run.smithi090.stdout:Total 162 kB/s | 31 kB 00:00 2024-09-13T15:24:08.024 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:24:08.027 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:24:08.027 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:24:08.037 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:08.047 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:24:08.047 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:24:08.135 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:24:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:08.303 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:08.303 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:08.303 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:24:08.303 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:08.303 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:08.303 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:08.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-mirror 2024-09-13T15:24:08.688 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:08.919 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:08.920 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:08.920 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:08.920 INFO:teuthology.orchestra.run.smithi090.stdout: libcephfs-devel-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:08.920 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:08.920 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:09.192 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:09.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librados2 2024-09-13T15:24:09.329 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:09.330 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:09.330 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repo Size 2024-09-13T15:24:09.330 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:09.330 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:24:09.330 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T15:24:09.330 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 3.0 M 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 12 M 2024-09-13T15:24:09.331 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:24:09.701 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-mirror-18.2.4-850.g50bbe7f3.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-09-13T15:24:09.702 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:09.703 INFO:teuthology.orchestra.run.smithi175.stdout:Total 8.0 MB/s | 3.0 MB 00:00 2024-09-13T15:24:09.703 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:24:09.711 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:24:09.712 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:24:09.761 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:38 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:09.791 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:24:09.791 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:24:09.849 INFO:teuthology.orchestra.run.smithi090.stdout:Package librados2-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:09.892 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:09.893 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:09.893 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:10.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install librbd1 2024-09-13T15:24:10.218 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:24:10.277 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:10.302 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:10.302 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:24:10.302 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T15:24:10.303 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T15:24:10.303 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T15:24:10.303 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:10.584 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:39 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:10.674 INFO:teuthology.orchestra.run.smithi090.stdout:Package librbd1-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:10.716 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:10.717 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:10.717 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:10.796 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rados 2024-09-13T15:24:11.361 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:40 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:11.452 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rados-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:11.485 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:11.486 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:11.486 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:24:11.486 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:11.486 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:11.486 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:11.494 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:11.495 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:11.495 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:11.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rgw 2024-09-13T15:24:11.738 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install rbd-nbd 2024-09-13T15:24:12.173 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:12.262 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rgw-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:12.304 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:12.305 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:12.305 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:12.321 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:12.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-cephfs 2024-09-13T15:24:12.457 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:24:12.457 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:12.457 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 172 k 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:Install 1 Package 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 172 k 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 494 k 2024-09-13T15:24:12.458 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:24:12.665 INFO:teuthology.orchestra.run.smithi175.stdout:rbd-nbd-18.2.4-850.g50bbe7f3.el9.x86_64.rpm 840 kB/s | 172 kB 00:00 2024-09-13T15:24:12.665 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:12.666 INFO:teuthology.orchestra.run.smithi175.stdout:Total 834 kB/s | 172 kB 00:00 2024-09-13T15:24:12.666 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:24:12.672 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:24:12.672 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:24:12.734 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:24:12.735 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:24:12.887 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:24:12.980 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:41 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:13.002 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:13.070 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-cephfs-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:13.114 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:13.115 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:13.115 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:13.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install python3-rbd 2024-09-13T15:24:13.386 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:13.661 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:13.661 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:13.661 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:24:13.661 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:13.661 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:24:13.661 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:24:13.800 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:42 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:13.891 DEBUG:teuthology.parallel:result is None 2024-09-13T15:24:13.891 INFO:teuthology.orchestra.run.smithi090.stdout:Package python3-rbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 is already installed. 2024-09-13T15:24:13.933 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:13.934 INFO:teuthology.orchestra.run.smithi090.stdout:Nothing to do. 2024-09-13T15:24:13.934 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:14.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-fuse 2024-09-13T15:24:14.590 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:43 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:14.724 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:14.724 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 87 k 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 87 k 2024-09-13T15:24:14.725 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 227 k 2024-09-13T15:24:14.726 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:24:14.920 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-fuse-18.2.4-850.g50bbe7f3.el9.x86_64.rpm 447 kB/s | 87 kB 00:00 2024-09-13T15:24:14.920 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:14.921 INFO:teuthology.orchestra.run.smithi090.stdout:Total 444 kB/s | 87 kB 00:00 2024-09-13T15:24:14.921 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:24:14.927 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:24:14.927 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:24:14.989 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:24:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:24:15.249 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:15.692 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:15.996 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:15.996 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:15.996 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:15.996 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-fuse-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:15.997 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:15.997 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:16.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-mirror 2024-09-13T15:24:16.961 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:45 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:17.092 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:17.093 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: Package Arch Version Repo Size 2024-09-13T15:24:17.093 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:17.093 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:24:17.093 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 3.0 M 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 3.0 M 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 12 M 2024-09-13T15:24:17.094 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:24:17.411 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-mirror-18.2.4-850.g50bbe7f3.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2024-09-13T15:24:17.412 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:17.412 INFO:teuthology.orchestra.run.smithi090.stdout:Total 9.4 MB/s | 3.0 MB 00:00 2024-09-13T15:24:17.412 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:24:17.421 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:24:17.421 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:24:17.501 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:24:17.501 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:24:17.924 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:24:17.996 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:18.021 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:18.021 INFO:teuthology.orchestra.run.smithi090.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T15:24:18.021 INFO:teuthology.orchestra.run.smithi090.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T15:24:18.021 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T15:24:18.021 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T15:24:18.022 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:19.192 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:19.192 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:19.192 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:19.193 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-mirror-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:19.193 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:19.193 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:19.521 DEBUG:teuthology.orchestra.run.smithi090:> sudo yum -y install rbd-nbd 2024-09-13T15:24:20.103 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:01:49 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:24:20.244 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd x86_64 2:18.2.4-850.g50bbe7f3.el9 ceph 172 k 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:24:20.245 INFO:teuthology.orchestra.run.smithi090.stdout:Install 1 Package 2024-09-13T15:24:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:20.246 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 172 k 2024-09-13T15:24:20.246 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 494 k 2024-09-13T15:24:20.246 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:24:20.463 INFO:teuthology.orchestra.run.smithi090.stdout:rbd-nbd-18.2.4-850.g50bbe7f3.el9.x86_64.rpm 793 kB/s | 172 kB 00:00 2024-09-13T15:24:20.464 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:20.464 INFO:teuthology.orchestra.run.smithi090.stdout:Total 786 kB/s | 172 kB 00:00 2024-09-13T15:24:20.465 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:24:20.472 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:24:20.472 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:24:20.538 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:24:20.539 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:24:20.693 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:24:20.803 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 1/1 2024-09-13T15:24:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:21.601 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:24:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: rbd-nbd-2:18.2.4-850.g50bbe7f3.el9.x86_64 2024-09-13T15:24:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:24:21.601 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:24:21.894 DEBUG:teuthology.parallel:result is None 2024-09-13T15:24:21.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:24:22.052 DEBUG:teuthology.orchestra.run.smithi090:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T15:24:22.073 INFO:teuthology.orchestra.run.smithi090.stdout:18.2.4-850.g50bbe7f3.el9 2024-09-13T15:24:22.074 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-850.g50bbe7f3.el9 2024-09-13T15:24:22.074 INFO:teuthology.task.install:The correct ceph version 18.2.4-850.g50bbe7f3 is installed. 2024-09-13T15:24:22.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:24:22.228 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T15:24:22.255 INFO:teuthology.orchestra.run.smithi175.stdout:18.2.4-850.g50bbe7f3.el9 2024-09-13T15:24:22.255 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-850.g50bbe7f3.el9 2024-09-13T15:24:22.255 INFO:teuthology.task.install:The correct ceph version 18.2.4-850.g50bbe7f3 is installed. 2024-09-13T15:24:22.257 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T15:24:22.257 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:24:22.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T15:24:22.282 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:24:22.282 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T15:24:22.325 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T15:24:22.325 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:24:22.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T15:24:22.350 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T15:24:22.414 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:24:22.414 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T15:24:22.442 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T15:24:22.509 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T15:24:22.510 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:24:22.510 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T15:24:22.534 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T15:24:22.627 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:24:22.627 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T15:24:22.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T15:24:22.722 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T15:24:22.722 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:24:22.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T15:24:22.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T15:24:22.812 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:24:22.813 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T15:24:22.840 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T15:24:22.907 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-13T15:24:22.997 INFO:tasks.cephadm:Config: {'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd op complaint time': 180, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'} 2024-09-13T15:24:22.997 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:24:22.997 INFO:tasks.cephadm:Cluster fsid is 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:24:22.997 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-13T15:24:22.997 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-09-13T15:24:22.998 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi090': '172.21.15.90', 'mon.smithi175': '172.21.15.175'} 2024-09-13T15:24:22.998 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-13T15:24:22.998 DEBUG:teuthology.orchestra.run.smithi090:> sudo hostname $(hostname -s) 2024-09-13T15:24:23.023 DEBUG:teuthology.orchestra.run.smithi175:> sudo hostname $(hostname -s) 2024-09-13T15:24:23.050 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-13T15:24:23.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:24:23.197 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82946/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.2.4-850-g50bbe7f3', 'node_name': '172.21.2.7+braggi07', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '18.2.4-850.g50bbe7f3'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-10 12:31:04.094992', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240910.095512-reef', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-13T15:24:23.339 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-rishabh-testing-20240910.095512-reef, sha1 50bbe7f395997dcd871813949c49addbbf630361 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:24:23.341 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/x86_64/flavors/default/cephadm 2024-09-13T15:24:23.341 DEBUG:teuthology.orchestra.run.smithi090:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T15:24:23.683 INFO:teuthology.orchestra.run.smithi090.stdout:-rw-r--r--. 1 ubuntu ubuntu 217224 Sep 13 15:24 /home/ubuntu/cephtest/cephadm 2024-09-13T15:24:23.684 DEBUG:teuthology.orchestra.run.smithi175:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-13T15:24:24.018 INFO:teuthology.orchestra.run.smithi175.stdout:-rw-r--r--. 1 ubuntu ubuntu 217224 Sep 13 15:24 /home/ubuntu/cephtest/cephadm 2024-09-13T15:24:24.018 DEBUG:teuthology.orchestra.run.smithi090:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T15:24:24.037 DEBUG:teuthology.orchestra.run.smithi175:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-13T15:24:24.066 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 on all hosts... 2024-09-13T15:24:24.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 pull 2024-09-13T15:24:24.080 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 pull 2024-09-13T15:24:25.641 INFO:teuthology.orchestra.run.smithi175.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361... 2024-09-13T15:24:26.470 INFO:teuthology.orchestra.run.smithi090.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361... 2024-09-13T15:24:52.619 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-13T15:24:52.620 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T15:24:52.620 INFO:teuthology.orchestra.run.smithi175.stdout: "image_id": "152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c", 2024-09-13T15:24:52.620 INFO:teuthology.orchestra.run.smithi175.stdout: "repo_digests": [ 2024-09-13T15:24:52.620 INFO:teuthology.orchestra.run.smithi175.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a" 2024-09-13T15:24:52.620 INFO:teuthology.orchestra.run.smithi175.stdout: ] 2024-09-13T15:24:52.620 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-13T15:24:56.864 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-09-13T15:24:56.864 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T15:24:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "image_id": "152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c", 2024-09-13T15:24:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "repo_digests": [ 2024-09-13T15:24:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a" 2024-09-13T15:24:56.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2024-09-13T15:24:56.865 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-09-13T15:24:56.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph 2024-09-13T15:24:56.943 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph 2024-09-13T15:24:56.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /etc/ceph 2024-09-13T15:24:57.003 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /etc/ceph 2024-09-13T15:24:57.045 INFO:tasks.cephadm:Writing seed config... 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [client] debug client = 20 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-09-13T15:24:57.047 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-13T15:24:57.048 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-13T15:24:57.049 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-09-13T15:24:57.049 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-13T15:24:57.049 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:24:57.049 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-13T15:24:57.065 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 41124e1c-71e4-11ef-bceb-c7b262605968 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-09-13T15:24:57.065 DEBUG:teuthology.orchestra.run.smithi090:mon.smithi090> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090.service 2024-09-13T15:24:57.107 INFO:tasks.cephadm:Bootstrapping... 2024-09-13T15:24:57.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 -v bootstrap --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.90 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:24:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:24:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361', '-v', 'bootstrap', '--fsid', '41124e1c-71e4-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.90', '--skip-admin-label'] 2024-09-13T15:24:57.284 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 5.2.2 2024-09-13T15:24:57.285 INFO:teuthology.orchestra.run.smithi090.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-13T15:24:57.285 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying podman|docker is present... 2024-09-13T15:24:57.306 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 5.2.2 2024-09-13T15:24:57.307 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying lvm2 is present... 2024-09-13T15:24:57.307 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying time synchronization is in place... 2024-09-13T15:24:57.314 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T15:24:57.314 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T15:24:57.320 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T15:24:57.320 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-09-13T15:24:57.327 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-09-13T15:24:57.333 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-09-13T15:24:57.333 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-09-13T15:24:57.333 INFO:teuthology.orchestra.run.smithi090.stdout:Repeating the final host check... 2024-09-13T15:24:57.355 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 5.2.2 2024-09-13T15:24:57.355 INFO:teuthology.orchestra.run.smithi090.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-13T15:24:57.355 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl is present 2024-09-13T15:24:57.355 INFO:teuthology.orchestra.run.smithi090.stdout:lvcreate is present 2024-09-13T15:24:57.361 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-13T15:24:57.361 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-13T15:24:57.367 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-13T15:24:57.367 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-09-13T15:24:57.373 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout enabled 2024-09-13T15:24:57.380 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout active 2024-09-13T15:24:57.380 INFO:teuthology.orchestra.run.smithi090.stdout:Unit chronyd.service is enabled and running 2024-09-13T15:24:57.380 INFO:teuthology.orchestra.run.smithi090.stdout:Host looks OK 2024-09-13T15:24:57.380 INFO:teuthology.orchestra.run.smithi090.stdout:Cluster fsid: 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:24:57.380 INFO:teuthology.orchestra.run.smithi090.stdout:Acquiring lock 139957256287568 on /run/cephadm/41124e1c-71e4-11ef-bceb-c7b262605968.lock 2024-09-13T15:24:57.380 INFO:teuthology.orchestra.run.smithi090.stdout:Lock 139957256287568 acquired on /run/cephadm/41124e1c-71e4-11ef-bceb-c7b262605968.lock 2024-09-13T15:24:57.381 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 3300 ... 2024-09-13T15:24:57.381 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying IP 172.21.15.90 port 6789 ... 2024-09-13T15:24:57.381 INFO:teuthology.orchestra.run.smithi090.stdout:Base mon IP(s) is [172.21.15.90:3300, 172.21.15.90:6789], mon addrv is [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-09-13T15:24:57.384 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.90 metric 100 2024-09-13T15:24:57.384 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.90 metric 100 2024-09-13T15:24:57.386 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-13T15:24:57.386 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-13T15:24:57.386 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-13T15:24:57.389 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-13T15:24:57.389 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-13T15:24:57.389 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T15:24:57.389 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-13T15:24:57.389 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d767/64 scope link noprefixroute 2024-09-13T15:24:57.389 INFO:teuthology.orchestra.run.smithi090.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-13T15:24:57.390 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-09-13T15:24:57.390 INFO:teuthology.orchestra.run.smithi090.stdout:Mon IP `172.21.15.90` is in CIDR network `172.21.0.0/20` 2024-09-13T15:24:57.390 INFO:teuthology.orchestra.run.smithi090.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-13T15:24:57.390 INFO:teuthology.orchestra.run.smithi090.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-13T15:24:57.391 INFO:teuthology.orchestra.run.smithi090.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361... 2024-09-13T15:24:57.834 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stdout 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c 2024-09-13T15:24:57.834 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361... 2024-09-13T15:24:57.834 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Getting image source signatures 2024-09-13T15:24:57.834 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:3bf8bde7788a2a6694fefa78ba3da98c997a778335fd0748e145012f747287ae 2024-09-13T15:24:57.835 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-13T15:24:57.835 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Copying config sha256:152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c 2024-09-13T15:24:57.835 INFO:teuthology.orchestra.run.smithi090.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-13T15:24:58.733 INFO:teuthology.orchestra.run.smithi090.stdout:ceph: stdout ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable) 2024-09-13T15:24:58.733 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph version: ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable) 2024-09-13T15:24:58.733 INFO:teuthology.orchestra.run.smithi090.stdout:Extracting ceph user uid/gid from container image... 2024-09-13T15:24:59.625 INFO:teuthology.orchestra.run.smithi090.stdout:stat: stdout 167 167 2024-09-13T15:24:59.625 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial keys... 2024-09-13T15:25:00.516 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQBLWeRmYMitORAAAO/od4eMYwdLD8WV6NdvLA== 2024-09-13T15:25:01.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQBMWeRmKTyRMRAALF650tpZC3zLJY+DTIWeEQ== 2024-09-13T15:25:02.293 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph-authtool: stdout AQBNWeRmT2e2KhAANUayb2RsGSTp/3ATrnwA0g== 2024-09-13T15:25:02.294 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial monmap... 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool for smithi090 [v2:172.21.15.90:3300,v1:172.21.15.90:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = pacific 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: set fsid to 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-13T15:25:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:25:03.211 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mon... 2024-09-13T15:25:04.132 INFO:teuthology.orchestra.run.smithi090.stdout:create mon.smithi090 on 2024-09-13T15:25:04.326 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-13T15:25:04.468 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-13T15:25:04.659 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-41124e1c-71e4-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-41124e1c-71e4-11ef-bceb-c7b262605968.target. 2024-09-13T15:25:04.659 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-41124e1c-71e4-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-41124e1c-71e4-11ef-bceb-c7b262605968.target. 2024-09-13T15:25:04.862 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090 2024-09-13T15:25:04.862 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090.service: Unit ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090.service not loaded. 2024-09-13T15:25:05.015 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-41124e1c-71e4-11ef-bceb-c7b262605968.target.wants/ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090.service → /etc/systemd/system/ceph-41124e1c-71e4-11ef-bceb-c7b262605968@.service. 2024-09-13T15:25:05.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:05 smithi090 podman[27417]: 2024-09-13 15:25:05.479065843 +0000 UTC m=+0.329788133 container start bba13e78c2fec8a49d46cc3ba822073f11909395c51c585d384112f2ac9d5b97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:25:05.595 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T15:25:05.595 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-09-13T15:25:05.603 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T15:25:05.603 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-09-13T15:25:05.603 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-09-13T15:25:05.603 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T15:25:05.603 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon to start... 2024-09-13T15:25:05.604 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mon... 2024-09-13T15:25:05.826 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:05 smithi090 ceph-mon[27440]: mkfs 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:05.827 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:05 smithi090 ceph-mon[27440]: mon.smithi090 is new leader, mons smithi090 in quorum (ranks 0) 2024-09-13T15:25:05.827 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:05 smithi090 bash[27417]: bba13e78c2fec8a49d46cc3ba822073f11909395c51c585d384112f2ac9d5b97 2024-09-13T15:25:05.827 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:05 smithi090 systemd[1]: Started Ceph mon.smithi090 for 41124e1c-71e4-11ef-bceb-c7b262605968. 2024-09-13T15:25:06.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:06 smithi090 ceph-mon[27440]: mon.smithi090 is new leader, mons smithi090 in quorum (ranks 0) 2024-09-13T15:25:06.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:06 smithi090 ceph-mon[27440]: monmap e1: 1 mons at {smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T15:25:06.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:06 smithi090 ceph-mon[27440]: fsmap 2024-09-13T15:25:06.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:06 smithi090 ceph-mon[27440]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T15:25:06.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:06 smithi090 ceph-mon[27440]: mgrmap e1: no daemons active 2024-09-13T15:25:06.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:06 smithi090 ceph-mon[27440]: from='client.? 172.21.15.90:0/1706672506' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-13T15:25:06.674 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout cluster: 2024-09-13T15:25:06.680 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout id: 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout services: 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi090 (age 0.553873s) 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout data: 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-13T15:25:06.681 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout pgs: 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:05.999+0000 7f4f58cb4640 1 Processor -- start 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:05.999+0000 7f4f58cb4640 1 -- start start 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.000+0000 7f4f58cb4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.000+0000 7f4f58cb4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54104390 con 0x7f4f541039c0 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.000+0000 7f4f5259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.000+0000 7f4f5259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40262/0 (socket says 172.21.15.90:40262) 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.000+0000 7f4f5259c640 1 -- 172.21.15.90:0/620073018 learned_addr learned my addr 172.21.15.90:0/620073018 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.000+0000 7f4f5259c640 1 -- 172.21.15.90:0/620073018 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f54104bc0 con 0x7f4f541039c0 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.002+0000 7f4f5259c640 1 --2- 172.21.15.90:0/620073018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54103dc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009820 tx=0x7f4f4402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792d052710a836d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:06.682 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f5159a640 1 -- 172.21.15.90:0/620073018 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4402ad40 con 0x7f4f541039c0 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f5159a640 1 -- 172.21.15.90:0/620073018 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f44032440 con 0x7f4f541039c0 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f5159a640 1 -- 172.21.15.90:0/620073018 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f4402aea0 con 0x7f4f541039c0 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 -- 172.21.15.90:0/620073018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 msgr2=0x7f4f54103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 --2- 172.21.15.90:0/620073018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54103dc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4f44009820 tx=0x7f4f4402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 -- 172.21.15.90:0/620073018 shutdown_connections 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 --2- 172.21.15.90:0/620073018 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54103dc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 -- 172.21.15.90:0/620073018 >> 172.21.15.90:0/620073018 conn(0x7f4f540ff110 msgr2=0x7f4f54101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 -- 172.21.15.90:0/620073018 shutdown_connections 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.003+0000 7f4f58cb4640 1 -- 172.21.15.90:0/620073018 wait complete. 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f58cb4640 1 Processor -- start 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f58cb4640 1 -- start start 2024-09-13T15:25:06.683 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f58cb4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54077a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f58cb4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f54077f90 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f5259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54077a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f5259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54077a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40274/0 (socket says 172.21.15.90:40274) 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.004+0000 7f4f5259c640 1 -- 172.21.15.90:0/1706672506 learned_addr learned my addr 172.21.15.90:0/1706672506 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f5259c640 1 -- 172.21.15.90:0/1706672506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f440094d0 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f5259c640 1 --2- 172.21.15.90:0/1706672506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54077a50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4f440090b0 tx=0x7f4f440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f44032e70 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f440308d0 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5419aca0 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f44030bd0 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.005+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5419b140 con 0x7f4f541039c0 2024-09-13T15:25:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.006+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f4f44039040 con 0x7f4f541039c0 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.006+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4f4403b740 con 0x7f4f541039c0 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.006+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f54190580 con 0x7f4f541039c0 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.008+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f4f44037070 con 0x7f4f541039c0 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.067+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4f54105170 con 0x7f4f541039c0 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.068+0000 7f4f437fe640 1 -- 172.21.15.90:0/1706672506 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4f4403b9a0 con 0x7f4f541039c0 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.068+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 msgr2=0x7f4f54077a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.068+0000 7f4f58cb4640 1 --2- 172.21.15.90:0/1706672506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54077a50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4f440090b0 tx=0x7f4f440046c0 comp rx=0 tx=0).stop 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.068+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 shutdown_connections 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.068+0000 7f4f58cb4640 1 --2- 172.21.15.90:0/1706672506 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4f541039c0 0x7f4f54077a50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.068+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 >> 172.21.15.90:0/1706672506 conn(0x7f4f540ff110 msgr2=0x7f4f5418b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.069+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 shutdown_connections 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:06.069+0000 7f4f58cb4640 1 -- 172.21.15.90:0/1706672506 wait complete. 2024-09-13T15:25:06.685 INFO:teuthology.orchestra.run.smithi090.stdout:mon is available 2024-09-13T15:25:06.686 INFO:teuthology.orchestra.run.smithi090.stdout:Assimilating anything we can from ceph.conf... 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.90:3300,v1:172.21.15.90:6789] 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T15:25:07.706 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2fa1d640 1 Processor -- start 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2fa1d640 1 -- start start 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2fa1d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2fa1d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd281054d0 con 0x7fdd28104b00 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2d7b9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:07.707 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2d7b9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40278/0 (socket says 172.21.15.90:40278) 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2d7b9640 1 -- 172.21.15.90:0/3199271231 learned_addr learned my addr 172.21.15.90:0/3199271231 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.058+0000 7fdd2d7b9640 1 -- 172.21.15.90:0/3199271231 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd28105d00 con 0x7fdd28104b00 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.059+0000 7fdd2d7b9640 1 --2- 172.21.15.90:0/3199271231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28104f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009820 tx=0x7fdd2402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55838cc2890f9609 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.059+0000 7fdd1ffff640 1 -- 172.21.15.90:0/3199271231 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd2402ad40 con 0x7fdd28104b00 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.059+0000 7fdd1ffff640 1 -- 172.21.15.90:0/3199271231 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd24032440 con 0x7fdd28104b00 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.059+0000 7fdd1ffff640 1 -- 172.21.15.90:0/3199271231 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd2402aea0 con 0x7fdd28104b00 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/3199271231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 msgr2=0x7fdd28104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 --2- 172.21.15.90:0/3199271231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28104f00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fdd24009820 tx=0x7fdd2402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/3199271231 shutdown_connections 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 --2- 172.21.15.90:0/3199271231 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28104f00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/3199271231 >> 172.21.15.90:0/3199271231 conn(0x7fdd281002d0 msgr2=0x7fdd281026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:07.708 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/3199271231 shutdown_connections 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/3199271231 wait complete. 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 Processor -- start 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.060+0000 7fdd2fa1d640 1 -- start start 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2fa1d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2fa1d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2807a0b0 con 0x7fdd28104b00 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2d7b9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2d7b9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28079b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40280/0 (socket says 172.21.15.90:40280) 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2d7b9640 1 -- 172.21.15.90:0/47759196 learned_addr learned my addr 172.21.15.90:0/47759196 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2d7b9640 1 -- 172.21.15.90:0/47759196 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd240094d0 con 0x7fdd28104b00 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2d7b9640 1 --2- 172.21.15.90:0/47759196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28079b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdd240090b0 tx=0x7fdd240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd240308d0 con 0x7fdd28104b00 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd24030a30 con 0x7fdd28104b00 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/47759196 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd280782a0 con 0x7fdd28104b00 2024-09-13T15:25:07.709 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd24030d30 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.061+0000 7fdd2fa1d640 1 -- 172.21.15.90:0/47759196 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd28078740 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.062+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fdd24039040 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.062+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fdd2403b960 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.063+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd08005250 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.065+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fdd24037070 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.119+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdd080023f0 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fdd24046cf0 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd1e7fc640 1 -- 172.21.15.90:0/47759196 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+338 (secure 0 0 0) 0x7fdd24045c10 con 0x7fdd28104b00 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 msgr2=0x7fdd28079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd13fff640 1 --2- 172.21.15.90:0/47759196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28079b70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fdd240090b0 tx=0x7fdd240046c0 comp rx=0 tx=0).stop 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 shutdown_connections 2024-09-13T15:25:07.710 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd13fff640 1 --2- 172.21.15.90:0/47759196 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdd28104b00 0x7fdd28079b70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:07.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.123+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 >> 172.21.15.90:0/47759196 conn(0x7fdd281002d0 msgr2=0x7fdd2818c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:07.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.124+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 shutdown_connections 2024-09-13T15:25:07.711 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:07.124+0000 7fdd13fff640 1 -- 172.21.15.90:0/47759196 wait complete. 2024-09-13T15:25:07.711 INFO:teuthology.orchestra.run.smithi090.stdout:Generating new minimal ceph.conf... 2024-09-13T15:25:07.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:07 smithi090 ceph-mon[27440]: from='client.? 172.21.15.90:0/47759196' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T15:25:07.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:07 smithi090 ceph-mon[27440]: from='client.? 172.21.15.90:0/47759196' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-13T15:25:08.723 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:08 smithi090 ceph-mon[27440]: from='client.? 172.21.15.90:0/1049333171' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:25:08.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.107+0000 7ffa345e7640 1 Processor -- start 2024-09-13T15:25:08.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.107+0000 7ffa345e7640 1 -- start start 2024-09-13T15:25:08.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.108+0000 7ffa345e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.108+0000 7ffa345e7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c1054e0 con 0x7ffa2c104b10 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.108+0000 7ffa32383640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.108+0000 7ffa32383640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40286/0 (socket says 172.21.15.90:40286) 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.108+0000 7ffa32383640 1 -- 172.21.15.90:0/1139198219 learned_addr learned my addr 172.21.15.90:0/1139198219 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.108+0000 7ffa32383640 1 -- 172.21.15.90:0/1139198219 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c105d10 con 0x7ffa2c104b10 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.109+0000 7ffa32383640 1 --2- 172.21.15.90:0/1139198219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c104f10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009820 tx=0x7ffa2002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efb8f518f0f79ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.109+0000 7ffa31381640 1 -- 172.21.15.90:0/1139198219 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2002ad40 con 0x7ffa2c104b10 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.109+0000 7ffa31381640 1 -- 172.21.15.90:0/1139198219 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ffa200328b0 con 0x7ffa2c104b10 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.109+0000 7ffa31381640 1 -- 172.21.15.90:0/1139198219 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa200307a0 con 0x7ffa2c104b10 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 -- 172.21.15.90:0/1139198219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 msgr2=0x7ffa2c104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 --2- 172.21.15.90:0/1139198219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c104f10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ffa20009820 tx=0x7ffa2002a270 comp rx=0 tx=0).stop 2024-09-13T15:25:08.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 -- 172.21.15.90:0/1139198219 shutdown_connections 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 --2- 172.21.15.90:0/1139198219 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c104f10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 -- 172.21.15.90:0/1139198219 >> 172.21.15.90:0/1139198219 conn(0x7ffa2c1002e0 msgr2=0x7ffa2c102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 -- 172.21.15.90:0/1139198219 shutdown_connections 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.110+0000 7ffa345e7640 1 -- 172.21.15.90:0/1139198219 wait complete. 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa345e7640 1 Processor -- start 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa345e7640 1 -- start start 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa345e7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa345e7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c198db0 con 0x7ffa2c104b10 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa32383640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa32383640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40288/0 (socket says 172.21.15.90:40288) 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa32383640 1 -- 172.21.15.90:0/1049333171 learned_addr learned my addr 172.21.15.90:0/1049333171 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:08.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa32383640 1 -- 172.21.15.90:0/1049333171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200094d0 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.111+0000 7ffa32383640 1 --2- 172.21.15.90:0/1049333171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c198870 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffa200097f0 tx=0x7ffa20030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa20040070 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ffa20032c30 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c106b90 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa2003a560 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c1992c0 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ffa2003aad0 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.112+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ffa20045c20 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.113+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2c199500 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.115+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7ffa20037070 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.171+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ffa2c108fc0 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.171+0000 7ffa1f7fe640 1 -- 172.21.15.90:0/1049333171 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7ffa20032da0 con 0x7ffa2c104b10 2024-09-13T15:25:08.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 msgr2=0x7ffa2c198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 --2- 172.21.15.90:0/1049333171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c198870 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ffa200097f0 tx=0x7ffa20030d60 comp rx=0 tx=0).stop 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 shutdown_connections 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 --2- 172.21.15.90:0/1049333171 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffa2c104b10 0x7ffa2c198870 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 >> 172.21.15.90:0/1049333171 conn(0x7ffa2c1002e0 msgr2=0x7ffa2c100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 shutdown_connections 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:08.172+0000 7ffa345e7640 1 -- 172.21.15.90:0/1049333171 wait complete. 2024-09-13T15:25:08.733 INFO:teuthology.orchestra.run.smithi090.stdout:Restarting the monitor... 2024-09-13T15:25:08.994 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:08 smithi090 systemd[1]: Stopping Ceph mon.smithi090 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:25:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:08 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090[27432]: 2024-09-13T15:25:08.991+0000 7f05ad2af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi090 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:25:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:08 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090[27432]: 2024-09-13T15:25:08.991+0000 7f05ad2af640 -1 mon.smithi090@0(leader) e1 *** Got Signal Terminated *** 2024-09-13T15:25:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 podman[27688]: 2024-09-13 15:25:09.043400736 +0000 UTC m=+0.153317792 container died bba13e78c2fec8a49d46cc3ba822073f11909395c51c585d384112f2ac9d5b97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:25:09.537 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 podman[27688]: 2024-09-13 15:25:09.358945124 +0000 UTC m=+0.468862179 container cleanup bba13e78c2fec8a49d46cc3ba822073f11909395c51c585d384112f2ac9d5b97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-13T15:25:09.537 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 bash[27688]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090 2024-09-13T15:25:09.988 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 podman[27700]: 2024-09-13 15:25:09.536366485 +0000 UTC m=+0.496366220 container remove bba13e78c2fec8a49d46cc3ba822073f11909395c51c585d384112f2ac9d5b97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-13T15:25:09.988 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 systemd[1]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090.service: Deactivated successfully. 2024-09-13T15:25:09.988 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 systemd[1]: Stopped Ceph mon.smithi090 for 41124e1c-71e4-11ef-bceb-c7b262605968. 2024-09-13T15:25:09.988 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 systemd[1]: Starting Ceph mon.smithi090 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:25:10.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 podman[27792]: 2024-09-13 15:25:09.987388253 +0000 UTC m=+0.110316879 container create 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:25:10.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:09 smithi090 podman[27792]: 2024-09-13 15:25:09.892755382 +0000 UTC m=+0.015684015 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:25:10.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 podman[27792]: 2024-09-13 15:25:10.190052717 +0000 UTC m=+0.312981346 container init 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T15:25:10.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 podman[27792]: 2024-09-13 15:25:10.193528291 +0000 UTC m=+0.316456926 container start 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: set uid:gid to 167:167 (ceph:ceph) 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable), process ceph-mon, pid 2 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: pidfile_write: ignore empty --pid-file 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: load: jerasure load: lrc 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: RocksDB version: 7.9.2 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Git sha 0 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Compile date 2024-09-10 11:17:42 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: DB SUMMARY 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: DB Session ID: OHI21X22YG1X1NZLIYIJ 2024-09-13T15:25:10.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: CURRENT file: CURRENT 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: IDENTITY file: IDENTITY 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi090/store.db dir, Total Num: 1, files: 000008.sst 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi090/store.db: 000009.log size: 92546 ; 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.error_if_exists: 0 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.create_if_missing: 0 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.paranoid_checks: 1 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.env: 0x55a320d97bc0 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.fs: PosixFileSystem 2024-09-13T15:25:10.254 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.info_log: 0x55a322754dc0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_file_opening_threads: 16 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.statistics: (nil) 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.use_fsync: 0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_log_file_size: 0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.keep_log_file_num: 1000 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.recycle_log_file_num: 0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.allow_fallocate: 1 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.allow_mmap_reads: 0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.allow_mmap_writes: 0 2024-09-13T15:25:10.255 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.use_direct_reads: 0 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.create_missing_column_families: 0 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.db_log_dir: 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.wal_dir: 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.advise_random_on_open: 1 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.db_write_buffer_size: 0 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.write_buffer_manager: 0x55a322762320 2024-09-13T15:25:10.256 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-13T15:25:10.257 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-13T15:25:10.257 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-13T15:25:10.257 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.rate_limiter: (nil) 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.wal_recovery_mode: 2 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.enable_thread_tracking: 0 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.enable_pipelined_write: 0 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.unordered_write: 0 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.row_cache: None 2024-09-13T15:25:10.258 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.wal_filter: None 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.allow_ingest_behind: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.two_write_queues: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.manual_wal_flush: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.wal_compression: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.atomic_flush: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.log_readahead_size: 0 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-13T15:25:10.259 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.best_efforts_recovery: 0 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.allow_data_in_errors: 0 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.db_host_id: __hostname__ 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_background_jobs: 2 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_background_compactions: -1 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_subcompactions: 1 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-13T15:25:10.260 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_total_wal_size: 0 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_open_files: -1 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bytes_per_sync: 0 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_readahead_size: 0 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_background_flushes: -1 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Compression algorithms supported: 2024-09-13T15:25:10.261 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kZSTD supported: 0 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kXpressCompression supported: 0 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kBZip2Compression supported: 0 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kLZ4Compression supported: 1 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kZlibCompression supported: 1 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kLZ4HCCompression supported: 1 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: kSnappyCompression supported: 1 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000010 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.merge_operator: 2024-09-13T15:25:10.262 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_filter: None 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_filter_factory: None 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.sst_partitioner_factory: None 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a322754a20) 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: cache_index_and_filter_blocks: 1 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: pin_top_level_index_and_filter: 1 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: index_type: 0 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: data_block_index_type: 0 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: index_shortening: 1 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-13T15:25:10.263 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: checksum: 4 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: no_block_cache: 0 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache: 0x55a322749350 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache_name: BinnedLRUCache 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache_options: 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: capacity : 536870912 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: num_shard_bits : 4 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: strict_capacity_limit : 0 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: high_pri_pool_ratio: 0.000 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_cache_compressed: (nil) 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: persistent_cache: (nil) 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_size: 4096 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_size_deviation: 10 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_restart_interval: 16 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: index_block_restart_interval: 1 2024-09-13T15:25:10.264 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: metadata_block_size: 4096 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: partition_filters: 0 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: use_delta_encoding: 1 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: filter_policy: bloomfilter 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: whole_key_filtering: 1 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: verify_compression: 0 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: read_amp_bytes_per_bit: 0 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: format_version: 5 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: enable_index_compression: 1 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: block_align: 0 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: max_auto_readahead_size: 262144 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: prepopulate_block_cache: 0 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: initial_auto_readahead_size: 8192 2024-09-13T15:25:10.265 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: num_file_reads_for_auto_readahead: 2 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.write_buffer_size: 33554432 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_write_buffer_number: 2 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression: NoCompression 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression: Disabled 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.prefix_extractor: nullptr 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.num_levels: 7 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-13T15:25:10.266 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.level: 32767 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.strategy: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.enabled: false 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-13T15:25:10.267 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.target_file_size_base: 67108864 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-13T15:25:10.268 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.arena_block_size: 1048576 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.disable_auto_compactions: 0 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-13T15:25:10.269 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.inplace_update_support: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.bloom_locality: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.max_successive_merges: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-13T15:25:10.270 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.paranoid_file_checks: 0 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.force_consistency_checks: 1 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.report_bg_io_stats: 0 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.ttl: 2592000 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.enable_blob_files: false 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.min_blob_size: 0 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.blob_file_size: 268435456 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-13T15:25:10.271 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.blob_file_starting_level: 0 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi090/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 77142a25-5eb3-4544-8c85-c5078d505bac 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726241110219368, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-13T15:25:10.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726241110220702, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 88537, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 286, "table_properties": {"data_size": 86598, "index_size": 259, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11938, "raw_average_key_size": 48, "raw_value_size": 79926, "raw_average_value_size": 326, "num_data_blocks": 11, "num_entries": 245, "num_filter_entries": 245, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726241110, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "77142a25-5eb3-4544-8c85-c5078d505bac", "db_session_id": "OHI21X22YG1X1NZLIYIJ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726241110220845, "job": 1, "event": "recovery_finished"} 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi090/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a322774e00 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: DB pointer 0x55a322868000 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** DB Stats ** 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-13T15:25:10.273 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** Compaction Stats [default] ** 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: L0 2/0 88.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Sum 2/0 88.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 106.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-13T15:25:10.274 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** Compaction Stats [default] ** 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 106.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Cumulative compaction: 0.00 GB write, 19.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T15:25:10.275 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Interval compaction: 0.00 GB write, 19.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Block cache BinnedLRUCache@0x55a322749350#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: starting mon.smithi090 rank 0 at public addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] at bind addrs [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi090 fsid 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???) e1 preinit fsid 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).mds e1 new map 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).mds e1 print_map 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: e1 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: legacy client fscid: -1 2024-09-13T15:25:10.276 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: 2024-09-13T15:25:10.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: No filesystems configured 2024-09-13T15:25:10.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-13T15:25:10.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T15:25:10.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T15:25:10.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-13T15:25:10.277 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-13T15:25:10.293 INFO:teuthology.orchestra.run.smithi090.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-13T15:25:10.602 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mon.smithi090 is new leader, mons smithi090 in quorum (ranks 0) 2024-09-13T15:25:10.602 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: monmap e1: 1 mons at {smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T15:25:10.603 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: fsmap 2024-09-13T15:25:10.603 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: osdmap e1: 0 total, 0 up, 0 in 2024-09-13T15:25:10.603 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 ceph-mon[27811]: mgrmap e1: no daemons active 2024-09-13T15:25:10.603 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 bash[27792]: 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d 2024-09-13T15:25:10.603 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:10 smithi090 systemd[1]: Started Ceph mon.smithi090 for 41124e1c-71e4-11ef-bceb-c7b262605968. 2024-09-13T15:25:11.327 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.680+0000 7f00b6ea1640 1 Processor -- start 2024-09-13T15:25:11.331 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.680+0000 7f00b6ea1640 1 -- start start 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.680+0000 7f00b6ea1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b0104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.680+0000 7f00b6ea1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b01054d0 con 0x7f00b0104b00 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.680+0000 7f00b4c3d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b0104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.681+0000 7f00b4c3d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b0104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40296/0 (socket says 172.21.15.90:40296) 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.681+0000 7f00b4c3d640 1 -- 172.21.15.90:0/1625135970 learned_addr learned my addr 172.21.15.90:0/1625135970 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.681+0000 7f00b4c3d640 1 -- 172.21.15.90:0/1625135970 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b0105d00 con 0x7f00b0104b00 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.683+0000 7f00b4c3d640 1 --2- 172.21.15.90:0/1625135970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b0104f00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009820 tx=0x7f00a402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cdedde9a3ff3c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.683+0000 7f00af7fe640 1 -- 172.21.15.90:0/1625135970 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a402ad40 con 0x7f00b0104b00 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.683+0000 7f00af7fe640 1 -- 172.21.15.90:0/1625135970 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f00a40328b0 con 0x7f00b0104b00 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.683+0000 7f00af7fe640 1 -- 172.21.15.90:0/1625135970 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a40307a0 con 0x7f00b0104b00 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.683+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1625135970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 msgr2=0x7f00b0104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:11.332 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.683+0000 7f00b6ea1640 1 --2- 172.21.15.90:0/1625135970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b0104f00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009820 tx=0x7f00a402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1625135970 shutdown_connections 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 --2- 172.21.15.90:0/1625135970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b0104f00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1625135970 >> 172.21.15.90:0/1625135970 conn(0x7f00b01002b0 msgr2=0x7f00b01026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1625135970 shutdown_connections 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1625135970 wait complete. 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 Processor -- start 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 -- start start 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b01988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.684+0000 7f00b6ea1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00b0198de0 con 0x7f00b0104b00 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b4c3d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b01988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b4c3d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b01988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40304/0 (socket says 172.21.15.90:40304) 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b4c3d640 1 -- 172.21.15.90:0/1908056445 learned_addr learned my addr 172.21.15.90:0/1908056445 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:11.333 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b4c3d640 1 -- 172.21.15.90:0/1908056445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40094d0 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b4c3d640 1 --2- 172.21.15.90:0/1908056445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b01988a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009940 tx=0x7f00a4030ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a4032c30 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f00a4032d90 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00b0106b80 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00a403a560 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.685+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00b01992f0 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.686+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f00a403ab20 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.686+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f00a4046c30 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.686+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00b0199530 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.689+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f00a4037070 con 0x7f00b0104b00 2024-09-13T15:25:11.334 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.743+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f00b0108fb0 con 0x7f00b0104b00 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.746+0000 7f00adffb640 1 -- 172.21.15.90:0/1908056445 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f00a4046df0 con 0x7f00b0104b00 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 msgr2=0x7f00b01988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 --2- 172.21.15.90:0/1908056445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b01988a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009940 tx=0x7f00a4030ce0 comp rx=0 tx=0).stop 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 shutdown_connections 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 --2- 172.21.15.90:0/1908056445 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f00b0104b00 0x7f00b01988a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 >> 172.21.15.90:0/1908056445 conn(0x7f00b01002b0 msgr2=0x7f00b0100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 shutdown_connections 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:10.747+0000 7f00b6ea1640 1 -- 172.21.15.90:0/1908056445 wait complete. 2024-09-13T15:25:11.335 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-13T15:25:11.349 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:25:11.349 INFO:teuthology.orchestra.run.smithi090.stdout:Creating mgr... 2024-09-13T15:25:11.350 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-13T15:25:11.350 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-13T15:25:11.350 INFO:teuthology.orchestra.run.smithi090.stdout:Verifying port 0.0.0.0:8443 ... 2024-09-13T15:25:11.585 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mgr.smithi090.fyvvpd 2024-09-13T15:25:11.585 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Failed to reset failed state of unit ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mgr.smithi090.fyvvpd.service: Unit ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mgr.smithi090.fyvvpd.service not loaded. 2024-09-13T15:25:11.728 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-41124e1c-71e4-11ef-bceb-c7b262605968.target.wants/ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mgr.smithi090.fyvvpd.service → /etc/systemd/system/ceph-41124e1c-71e4-11ef-bceb-c7b262605968@.service. 2024-09-13T15:25:11.944 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:11 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1908056445' entity='client.admin' 2024-09-13T15:25:12.229 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T15:25:12.229 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-09-13T15:25:12.236 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T15:25:12.236 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-09-13T15:25:12.236 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-09-13T15:25:12.236 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to enable service . firewalld.service is not available 2024-09-13T15:25:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T15:25:12.245 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-09-13T15:25:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T15:25:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-09-13T15:25:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-09-13T15:25:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[9283, 8765, 8443]>. firewalld.service is not available 2024-09-13T15:25:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr to start... 2024-09-13T15:25:12.252 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr... 2024-09-13T15:25:13.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:12 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2891231596' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T15:25:13.430 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:13.430 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:25:13.430 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "41124e1c-71e4-11ef-bceb-c7b262605968", 2024-09-13T15:25:13.431 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T15:25:13.431 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T15:25:13.431 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T15:25:13.431 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T15:25:13.431 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.432 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T15:25:13.432 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T15:25:13.432 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-09-13T15:25:13.432 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "smithi090" 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T15:25:13.433 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T15:25:13.434 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T15:25:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T15:25:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T15:25:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T15:25:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T15:25:13.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T15:25:13.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T15:25:13.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T15:25:05.517057+0000", 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.644+0000 7fb6a4906640 1 Processor -- start 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb6a4906640 1 -- start start 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb6a4906640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a01079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb6a4906640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a0077e50 con 0x7fb6a01055c0 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a01079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a01079b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40324/0 (socket says 172.21.15.90:40324) 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb69e59c640 1 -- 172.21.15.90:0/3330652229 learned_addr learned my addr 172.21.15.90:0/3330652229 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:13.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.645+0000 7fb69e59c640 1 -- 172.21.15.90:0/3330652229 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0107ef0 con 0x7fb6a01055c0 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.646+0000 7fb69e59c640 1 --2- 172.21.15.90:0/3330652229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a01079b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb690009820 tx=0x7fb69002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a468c29321dc2d53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.646+0000 7fb69d59a640 1 -- 172.21.15.90:0/3330652229 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb69002ad40 con 0x7fb6a01055c0 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.646+0000 7fb69d59a640 1 -- 172.21.15.90:0/3330652229 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb6900328b0 con 0x7fb6a01055c0 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.646+0000 7fb69d59a640 1 -- 172.21.15.90:0/3330652229 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6900307a0 con 0x7fb6a01055c0 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 -- 172.21.15.90:0/3330652229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 msgr2=0x7fb6a01079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 --2- 172.21.15.90:0/3330652229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a01079b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb690009820 tx=0x7fb69002a270 comp rx=0 tx=0).stop 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 -- 172.21.15.90:0/3330652229 shutdown_connections 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 --2- 172.21.15.90:0/3330652229 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a01079b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 -- 172.21.15.90:0/3330652229 >> 172.21.15.90:0/3330652229 conn(0x7fb6a00ff200 msgr2=0x7fb6a0101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 -- 172.21.15.90:0/3330652229 shutdown_connections 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.647+0000 7fb6a4906640 1 -- 172.21.15.90:0/3330652229 wait complete. 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb6a4906640 1 Processor -- start 2024-09-13T15:25:13.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb6a4906640 1 -- start start 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb6a4906640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a0104e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb6a4906640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a0105380 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a0104e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb69e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a0104e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:40334/0 (socket says 172.21.15.90:40334) 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.648+0000 7fb69e59c640 1 -- 172.21.15.90:0/2891231596 learned_addr learned my addr 172.21.15.90:0/2891231596 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb69e59c640 1 -- 172.21.15.90:0/2891231596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6900094d0 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb69e59c640 1 --2- 172.21.15.90:0/2891231596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a0104e40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb690030490 tx=0x7fb690032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb690030eb0 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb690032c30 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a0103570 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb69003a560 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a0103a10 con 0x7fb6a01055c0 2024-09-13T15:25:13.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fb69003aad0 con 0x7fb6a01055c0 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.649+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb690046c70 con 0x7fb6a01055c0 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.650+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a0105a20 con 0x7fb6a01055c0 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.652+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fb690037070 con 0x7fb6a01055c0 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.722+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb6a0108ee0 con 0x7fb6a01055c0 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.723+0000 7fb68f7fe640 1 -- 172.21.15.90:0/2891231596 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7fb690032da0 con 0x7fb6a01055c0 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 msgr2=0x7fb6a0104e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 --2- 172.21.15.90:0/2891231596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a0104e40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb690030490 tx=0x7fb690032840 comp rx=0 tx=0).stop 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 shutdown_connections 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 --2- 172.21.15.90:0/2891231596 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6a01055c0 0x7fb6a0104e40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 >> 172.21.15.90:0/2891231596 conn(0x7fb6a00ff200 msgr2=0x7fb6a00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 shutdown_connections 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:12.724+0000 7fb6a4906640 1 -- 172.21.15.90:0/2891231596 wait complete. 2024-09-13T15:25:13.447 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (1/15)... 2024-09-13T15:25:16.125 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2806542927' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T15:25:16.538 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:16.538 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:25:16.538 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "41124e1c-71e4-11ef-bceb-c7b262605968", 2024-09-13T15:25:16.538 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "smithi090" 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T15:25:16.540 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T15:25:16.541 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T15:25:05.517057+0000", 2024-09-13T15:25:16.542 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.835+0000 7f82ff247640 1 Processor -- start 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.835+0000 7f82ff247640 1 -- start start 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.835+0000 7f82ff247640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.835+0000 7f82ff247640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8104390 con 0x7f82f81039c0 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.836+0000 7f82fcfe3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.836+0000 7f82fcfe3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41138/0 (socket says 172.21.15.90:41138) 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.836+0000 7f82fcfe3640 1 -- 172.21.15.90:0/3016580271 learned_addr learned my addr 172.21.15.90:0/3016580271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.836+0000 7f82fcfe3640 1 -- 172.21.15.90:0/3016580271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f8104bc0 con 0x7f82f81039c0 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.837+0000 7f82fcfe3640 1 --2- 172.21.15.90:0/3016580271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8103dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009820 tx=0x7f82ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb8640a2c7e59a32 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.837+0000 7f82f77fe640 1 -- 172.21.15.90:0/3016580271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ec02ad40 con 0x7f82f81039c0 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.837+0000 7f82f77fe640 1 -- 172.21.15.90:0/3016580271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f82ec0328b0 con 0x7f82f81039c0 2024-09-13T15:25:16.543 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.837+0000 7f82f77fe640 1 -- 172.21.15.90:0/3016580271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ec0307a0 con 0x7f82f81039c0 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 -- 172.21.15.90:0/3016580271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 msgr2=0x7f82f8103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 --2- 172.21.15.90:0/3016580271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8103dc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009820 tx=0x7f82ec02a270 comp rx=0 tx=0).stop 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 -- 172.21.15.90:0/3016580271 shutdown_connections 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 --2- 172.21.15.90:0/3016580271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8103dc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 -- 172.21.15.90:0/3016580271 >> 172.21.15.90:0/3016580271 conn(0x7f82f80ff110 msgr2=0x7f82f8101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 -- 172.21.15.90:0/3016580271 shutdown_connections 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 -- 172.21.15.90:0/3016580271 wait complete. 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.838+0000 7f82ff247640 1 Processor -- start 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82ff247640 1 -- start start 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82ff247640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82ff247640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f8198c80 con 0x7f82f81039c0 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82fcfe3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82fcfe3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41142/0 (socket says 172.21.15.90:41142) 2024-09-13T15:25:16.544 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82fcfe3640 1 -- 172.21.15.90:0/2806542927 learned_addr learned my addr 172.21.15.90:0/2806542927 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82fcfe3640 1 -- 172.21.15.90:0/2806542927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ec0094d0 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82fcfe3640 1 --2- 172.21.15.90:0/2806542927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8198740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0097f0 tx=0x7f82ec032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ec040070 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f82ec032c30 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.839+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82ec03a560 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.840+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f8198e80 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.840+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f8199320 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.840+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f82ec03a9c0 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.840+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f82ec045b80 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.841+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82f8103dc0 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.843+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f82ec037070 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.902+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f82f80770c0 con 0x7f82f81039c0 2024-09-13T15:25:16.545 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.903+0000 7f82f5ffb640 1 -- 172.21.15.90:0/2806542927 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1246 (secure 0 0 0) 0x7f82ec032da0 con 0x7f82f81039c0 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 msgr2=0x7f82f8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 --2- 172.21.15.90:0/2806542927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8198740 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f82ec0097f0 tx=0x7f82ec032840 comp rx=0 tx=0).stop 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 shutdown_connections 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 --2- 172.21.15.90:0/2806542927 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82f81039c0 0x7f82f8198740 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 >> 172.21.15.90:0/2806542927 conn(0x7f82f80ff110 msgr2=0x7f82f80ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 shutdown_connections 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:15.904+0000 7f82ff247640 1 -- 172.21.15.90:0/2806542927 wait complete. 2024-09-13T15:25:16.546 INFO:teuthology.orchestra.run.smithi090.stdout:mgr not available, waiting (2/15)... 2024-09-13T15:25:17.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: Activating manager daemon smithi090.fyvvpd 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: mgrmap e2: smithi090.fyvvpd(active, starting, since 0.00275898s) 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr metadata", "who": "smithi090.fyvvpd", "id": "smithi090.fyvvpd"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: Manager daemon smithi090.fyvvpd is now available 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/mirror_snapshot_schedule"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/trash_purge_schedule"}]: dispatch 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:17.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:17.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:16 smithi090 ceph-mon[27811]: from='mgr.14100 172.21.15.90:0/2371636862' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:19.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:18 smithi090 ceph-mon[27811]: mgrmap e3: smithi090.fyvvpd(active, since 1.00699s) 2024-09-13T15:25:19.722 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:19.722 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsid": "41124e1c-71e4-11ef-bceb-c7b262605968", 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "health": { 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 0 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "smithi090" 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:19.723 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-13T15:25:19.724 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-13T15:25:19.725 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.725 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-13T15:25:19.725 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-13T15:25:19.725 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.726 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-13T15:25:19.728 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "restful" 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ], 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "modified": "2024-09-13T15:25:05.517057+0000", 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout }, 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.940+0000 7fd8c3658640 1 Processor -- start 2024-09-13T15:25:19.729 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.940+0000 7fd8c3658640 1 -- start start 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.941+0000 7fd8c3658640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc104ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.941+0000 7fd8c3658640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc1054c0 con 0x7fd8bc104af0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.941+0000 7fd8c13f4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc104ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.941+0000 7fd8c13f4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc104ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41200/0 (socket says 172.21.15.90:41200) 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.941+0000 7fd8c13f4640 1 -- 172.21.15.90:0/7389491 learned_addr learned my addr 172.21.15.90:0/7389491 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.941+0000 7fd8c13f4640 1 -- 172.21.15.90:0/7389491 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc105cf0 con 0x7fd8bc104af0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.942+0000 7fd8c13f4640 1 --2- 172.21.15.90:0/7389491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc104ef0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009820 tx=0x7fd8b402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d197aa1b31e66448 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.942+0000 7fd8bbfff640 1 -- 172.21.15.90:0/7389491 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b402ad40 con 0x7fd8bc104af0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.942+0000 7fd8bbfff640 1 -- 172.21.15.90:0/7389491 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd8b40328b0 con 0x7fd8bc104af0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.942+0000 7fd8bbfff640 1 -- 172.21.15.90:0/7389491 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b40307a0 con 0x7fd8bc104af0 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- 172.21.15.90:0/7389491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 msgr2=0x7fd8bc104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 --2- 172.21.15.90:0/7389491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc104ef0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009820 tx=0x7fd8b402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:19.730 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- 172.21.15.90:0/7389491 shutdown_connections 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 --2- 172.21.15.90:0/7389491 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc104ef0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- 172.21.15.90:0/7389491 >> 172.21.15.90:0/7389491 conn(0x7fd8bc100260 msgr2=0x7fd8bc1026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- 172.21.15.90:0/7389491 shutdown_connections 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- 172.21.15.90:0/7389491 wait complete. 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 Processor -- start 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- start start 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.943+0000 7fd8c3658640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc198e00 con 0x7fd8bc104af0 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c13f4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:19.731 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c13f4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc1988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41216/0 (socket says 172.21.15.90:41216) 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c13f4640 1 -- 172.21.15.90:0/320915207 learned_addr learned my addr 172.21.15.90:0/320915207 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c13f4640 1 -- 172.21.15.90:0/320915207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40094d0 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c13f4640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc1988c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40097f0 tx=0x7fd8b4032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b4040070 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd8b4032c30 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8b403a560 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc106b70 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.944+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc199310 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.945+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7fd8b403a6c0 con 0x7fd8bc104af0 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.945+0000 7fd8ba7fc640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7fd8a4038710 0x7fd8a403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:19.732 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.945+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc199550 con 0x7fd8bc104af0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.945+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd8b406b0f0 con 0x7fd8bc104af0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.945+0000 7fd8c0bf3640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7fd8a4038710 0x7fd8a403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.946+0000 7fd8c0bf3640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7fd8a4038710 0x7fd8a403abd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00098c0 tx=0x7fd8b0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:18.950+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd8b4043310 con 0x7fd8bc104af0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.172+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd8bc104ff0 con 0x7fd8bc104af0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.172+0000 7fd8ba7fc640 1 -- 172.21.15.90:0/320915207 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1245 (secure 0 0 0) 0x7fd8b4043310 con 0x7fd8bc104af0 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7fd8a4038710 msgr2=0x7fd8a403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7fd8a4038710 0x7fd8a403abd0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00098c0 tx=0x7fd8b0006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 msgr2=0x7fd8bc1988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc1988c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd8b40097f0 tx=0x7fd8b4032840 comp rx=0 tx=0).stop 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 shutdown_connections 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7fd8a4038710 0x7fd8a403abd0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:19.733 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 --2- 172.21.15.90:0/320915207 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8bc104af0 0x7fd8bc1988c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:19.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.175+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 >> 172.21.15.90:0/320915207 conn(0x7fd8bc100260 msgr2=0x7fd8bc100e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:19.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.176+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 shutdown_connections 2024-09-13T15:25:19.734 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:19.176+0000 7fd8c3658640 1 -- 172.21.15.90:0/320915207 wait complete. 2024-09-13T15:25:19.734 INFO:teuthology.orchestra.run.smithi090.stdout:mgr is available 2024-09-13T15:25:20.104 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:19 smithi090 ceph-mon[27811]: mgrmap e4: smithi090.fyvvpd(active, since 2s) 2024-09-13T15:25:20.104 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:19 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/320915207' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-13T15:25:20.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:20.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [global] 2024-09-13T15:25:20.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout fsid = 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:20.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout [osd] 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.102+0000 7f3fdf462640 1 Processor -- start 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.102+0000 7f3fdf462640 1 -- start start 2024-09-13T15:25:20.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.103+0000 7f3fdf462640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd810f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.103+0000 7f3fdf462640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd810fee0 con 0x7f3fd810f510 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.103+0000 7f3fdd1fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd810f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.103+0000 7f3fdd1fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd810f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41220/0 (socket says 172.21.15.90:41220) 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.103+0000 7f3fdd1fe640 1 -- 172.21.15.90:0/94315554 learned_addr learned my addr 172.21.15.90:0/94315554 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.103+0000 7f3fdd1fe640 1 -- 172.21.15.90:0/94315554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8110710 con 0x7f3fd810f510 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.104+0000 7f3fdd1fe640 1 --2- 172.21.15.90:0/94315554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd810f910 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009e80 tx=0x7f3fd402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535b44854c737b31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.104+0000 7f3fcffff640 1 -- 172.21.15.90:0/94315554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd4031400 con 0x7f3fd810f510 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.104+0000 7f3fcffff640 1 -- 172.21.15.90:0/94315554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3fd402d050 con 0x7f3fd810f510 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.104+0000 7f3fcffff640 1 -- 172.21.15.90:0/94315554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd40077c0 con 0x7f3fd810f510 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- 172.21.15.90:0/94315554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 msgr2=0x7f3fd810f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 --2- 172.21.15.90:0/94315554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd810f910 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3fd4009e80 tx=0x7f3fd402a7a0 comp rx=0 tx=0).stop 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- 172.21.15.90:0/94315554 shutdown_connections 2024-09-13T15:25:20.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 --2- 172.21.15.90:0/94315554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd810f910 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- 172.21.15.90:0/94315554 >> 172.21.15.90:0/94315554 conn(0x7f3fd806c0f0 msgr2=0x7f3fd806c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- 172.21.15.90:0/94315554 shutdown_connections 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- 172.21.15.90:0/94315554 wait complete. 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 Processor -- start 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- start start 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd81a32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.105+0000 7f3fdf462640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd81a3820 con 0x7f3fd810f510 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdd1fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd81a32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdd1fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd81a32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41224/0 (socket says 172.21.15.90:41224) 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdd1fe640 1 -- 172.21.15.90:0/2173510792 learned_addr learned my addr 172.21.15.90:0/2173510792 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:20.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdd1fe640 1 -- 172.21.15.90:0/2173510792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd4009b30 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdd1fe640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd81a32e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3fd402acc0 tx=0x7f3fd4031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd4031bd0 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3fd4031d30 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd8111590 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fd4039620 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.106+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd81a3d30 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.107+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7f3fd4038040 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.107+0000 7f3fce7fc640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f3fc40386c0 0x7f3fc403ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.107+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3fd406b150 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.107+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd810f990 con 0x7f3fd810f510 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.107+0000 7f3fdc9fd640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f3fc40386c0 0x7f3fc403ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:20.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.108+0000 7f3fdc9fd640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f3fc40386c0 0x7f3fc403ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3fc80098c0 tx=0x7f3fc8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.112+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f3fd4041030 con 0x7f3fd810f510 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.275+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f3fd8113940 con 0x7f3fd810f510 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.276+0000 7f3fce7fc640 1 -- 172.21.15.90:0/2173510792 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f3fd4069130 con 0x7f3fd810f510 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f3fc40386c0 msgr2=0x7f3fc403ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f3fc40386c0 0x7f3fc403ab80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3fc80098c0 tx=0x7f3fc8006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 msgr2=0x7f3fd81a32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd81a32e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3fd402acc0 tx=0x7f3fd4031630 comp rx=0 tx=0).stop 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 shutdown_connections 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f3fc40386c0 0x7f3fc403ab80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 --2- 172.21.15.90:0/2173510792 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fd810f510 0x7f3fd81a32e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 >> 172.21.15.90:0/2173510792 conn(0x7f3fd806c0f0 msgr2=0x7f3fd810ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 shutdown_connections 2024-09-13T15:25:20.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:20.278+0000 7f3fdf462640 1 -- 172.21.15.90:0/2173510792 wait complete. 2024-09-13T15:25:20.875 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling cephadm module... 2024-09-13T15:25:21.171 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:20 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2173510792' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-13T15:25:22.233 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:21 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/4246705775' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-13T15:25:22.572 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.248+0000 7ff18a8b0640 1 Processor -- start 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.248+0000 7ff18a8b0640 1 -- start start 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.249+0000 7ff18a8b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184104ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.249+0000 7ff18a8b0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1841054b0 con 0x7ff184104ae0 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.249+0000 7ff183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184104ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.249+0000 7ff183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184104ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41232/0 (socket says 172.21.15.90:41232) 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.249+0000 7ff183fff640 1 -- 172.21.15.90:0/2861867612 learned_addr learned my addr 172.21.15.90:0/2861867612 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.249+0000 7ff183fff640 1 -- 172.21.15.90:0/2861867612 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff184105ce0 con 0x7ff184104ae0 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff183fff640 1 --2- 172.21.15.90:0/2861867612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184104ee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff174009820 tx=0x7ff17402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c45eb9213c38582 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:22.577 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff182ffd640 1 -- 172.21.15.90:0/2861867612 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff17402ad40 con 0x7ff184104ae0 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff182ffd640 1 -- 172.21.15.90:0/2861867612 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff1740328b0 con 0x7ff184104ae0 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff182ffd640 1 -- 172.21.15.90:0/2861867612 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1740307a0 con 0x7ff184104ae0 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff18a8b0640 1 -- 172.21.15.90:0/2861867612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 msgr2=0x7ff184104ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff18a8b0640 1 --2- 172.21.15.90:0/2861867612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184104ee0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff174009820 tx=0x7ff17402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.250+0000 7ff18a8b0640 1 -- 172.21.15.90:0/2861867612 shutdown_connections 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 --2- 172.21.15.90:0/2861867612 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184104ee0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 -- 172.21.15.90:0/2861867612 >> 172.21.15.90:0/2861867612 conn(0x7ff1841002b0 msgr2=0x7ff1841026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 -- 172.21.15.90:0/2861867612 shutdown_connections 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 -- 172.21.15.90:0/2861867612 wait complete. 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 Processor -- start 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 -- start start 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:22.578 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.251+0000 7ff18a8b0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18407a0b0 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff183fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184079b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41246/0 (socket says 172.21.15.90:41246) 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff183fff640 1 -- 172.21.15.90:0/4246705775 learned_addr learned my addr 172.21.15.90:0/4246705775 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff183fff640 1 -- 172.21.15.90:0/4246705775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1740094d0 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff183fff640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184079b70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff174030490 tx=0x7ff174032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff174030e80 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff174032c30 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff184106b60 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff17403a560 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.252+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1840784f0 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.253+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 4) v1 ==== 45545+0+0 (secure 0 0 0) 0x7ff17403a6c0 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.253+0000 7ff1817fa640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7ff16c038710 0x7ff16c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.253+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff17406b550 con 0x7ff184104ae0 2024-09-13T15:25:22.579 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.253+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184108f10 con 0x7ff184104ae0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.253+0000 7ff1837fe640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7ff16c038710 0x7ff16c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.254+0000 7ff1837fe640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7ff16c038710 0x7ff16c03abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1780098c0 tx=0x7ff178006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.258+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7ff174046460 con 0x7ff184104ae0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.462+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7ff184078730 con 0x7ff184104ae0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.928+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7ff174041cc0 con 0x7ff184104ae0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.929+0000 7ff1817fa640 1 -- 172.21.15.90:0/4246705775 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7ff17406b2b0 con 0x7ff184104ae0 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7ff16c038710 msgr2=0x7ff16c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7ff16c038710 0x7ff16c03abd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1780098c0 tx=0x7ff178006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 msgr2=0x7ff184079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184079b70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff174030490 tx=0x7ff174032840 comp rx=0 tx=0).stop 2024-09-13T15:25:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 shutdown_connections 2024-09-13T15:25:22.581 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7ff16c038710 0x7ff16c03abd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:22.581 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 --2- 172.21.15.90:0/4246705775 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff184104ae0 0x7ff184079b70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:22.581 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 >> 172.21.15.90:0/4246705775 conn(0x7ff1841002b0 msgr2=0x7ff184100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:22.581 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 shutdown_connections 2024-09-13T15:25:22.581 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:21.931+0000 7ff18a8b0640 1 -- 172.21.15.90:0/4246705775 wait complete. 2024-09-13T15:25:23.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:22 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/4246705775' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-13T15:25:23.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:22 smithi090 ceph-mon[27811]: mgrmap e5: smithi090.fyvvpd(active, since 5s) 2024-09-13T15:25:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:23 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/734275692' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T15:25:27.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: Active manager daemon smithi090.fyvvpd restarted 2024-09-13T15:25:27.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: Activating manager daemon smithi090.fyvvpd 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: osdmap e2: 0 total, 0 up, 0 in 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: mgrmap e6: smithi090.fyvvpd(active, starting, since 0.00314785s) 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr metadata", "who": "smithi090.fyvvpd", "id": "smithi090.fyvvpd"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: Manager daemon smithi090.fyvvpd is now available 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:25:27.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/mirror_snapshot_schedule"}]: dispatch 2024-09-13T15:25:27.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:26 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/trash_purge_schedule"}]: dispatch 2024-09-13T15:25:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:27 smithi090 ceph-mon[27811]: Found migration_current of "None". Setting to last migration. 2024-09-13T15:25:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:27 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:27 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:27 smithi090 ceph-mon[27811]: mgrmap e7: smithi090.fyvvpd(active, since 1.00471s) 2024-09-13T15:25:29.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:28 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "smithi090.fyvvpd", 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f774239f640 1 Processor -- start 2024-09-13T15:25:29.021 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f774239f640 1 -- start start 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f774239f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c107a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f774239f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c077df0 con 0x7f773c105650 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f773bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c107a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f773bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c107a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41276/0 (socket says 172.21.15.90:41276) 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.964+0000 7f773bfff640 1 -- 172.21.15.90:0/3303666388 learned_addr learned my addr 172.21.15.90:0/3303666388 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.965+0000 7f773bfff640 1 -- 172.21.15.90:0/3303666388 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c077f30 con 0x7f773c105650 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.965+0000 7f773bfff640 1 --2- 172.21.15.90:0/3303666388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c107a40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f772c009820 tx=0x7f772c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8ebcde0de871509 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f773affd640 1 -- 172.21.15.90:0/3303666388 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c02ad40 con 0x7f773c105650 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f773affd640 1 -- 172.21.15.90:0/3303666388 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f772c0328b0 con 0x7f773c105650 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f773affd640 1 -- 172.21.15.90:0/3303666388 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c0307a0 con 0x7f773c105650 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 -- 172.21.15.90:0/3303666388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 msgr2=0x7f773c107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:29.022 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 --2- 172.21.15.90:0/3303666388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c107a40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f772c009820 tx=0x7f772c02a270 comp rx=0 tx=0).stop 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 -- 172.21.15.90:0/3303666388 shutdown_connections 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 --2- 172.21.15.90:0/3303666388 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c107a40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 -- 172.21.15.90:0/3303666388 >> 172.21.15.90:0/3303666388 conn(0x7f773c0ff230 msgr2=0x7f773c101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 -- 172.21.15.90:0/3303666388 shutdown_connections 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.966+0000 7f774239f640 1 -- 172.21.15.90:0/3303666388 wait complete. 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f774239f640 1 Processor -- start 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f774239f640 1 -- start start 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f774239f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f774239f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c1969f0 con 0x7f773c105650 2024-09-13T15:25:29.023 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f773bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f773bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c1964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41292/0 (socket says 172.21.15.90:41292) 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.967+0000 7f773bfff640 1 -- 172.21.15.90:0/734275692 learned_addr learned my addr 172.21.15.90:0/734275692 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f773bfff640 1 -- 172.21.15.90:0/734275692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c0094d0 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f773bfff640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c1964b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f772c030490 tx=0x7f772c032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c030eb0 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f772c032c30 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c196bf0 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f772c03a560 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.968+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c197090 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.969+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 5) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f772c03a7e0 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.969+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c1039f0 con 0x7f773c105650 2024-09-13T15:25:29.024 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.969+0000 7f77397fa640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 0x7f772403f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.969+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f772c06b6e0 con 0x7f773c105650 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.969+0000 7f773b7fe640 1 -- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 msgr2=0x7f772403f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1322912193 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.970+0000 7f773b7fe640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 0x7f772403f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:22.978+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f772c039d50 con 0x7f773c105650 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.171+0000 7f773b7fe640 1 -- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 msgr2=0x7f772403f020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/1322912193 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.171+0000 7f773b7fe640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 0x7f772403f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.194+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f773c18fb50 con 0x7f773c105650 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.195+0000 7f77397fa640 1 -- 172.21.15.90:0/734275692 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f772c044e90 con 0x7f773c105650 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 msgr2=0x7f772403f020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 0x7f772403f020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:29.025 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 msgr2=0x7f773c1964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c1964b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f772c030490 tx=0x7f772c032840 comp rx=0 tx=0).stop 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 shutdown_connections 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:6800/1322912193,v1:172.21.15.90:6801/1322912193] conn(0x7f772403cb60 0x7f772403f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 --2- 172.21.15.90:0/734275692 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f773c105650 0x7f773c1964b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 >> 172.21.15.90:0/734275692 conn(0x7f773c0ff230 msgr2=0x7f773c101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 shutdown_connections 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:23.197+0000 7f774239f640 1 -- 172.21.15.90:0/734275692 wait complete. 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-09-13T15:25:29.026 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 5... 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:29 smithi090 ceph-mon[27811]: [13/Sep/2024:15:25:28] ENGINE Bus STARTING 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:29 smithi090 ceph-mon[27811]: [13/Sep/2024:15:25:28] ENGINE Serving on http://172.21.15.90:8765 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:29 smithi090 ceph-mon[27811]: [13/Sep/2024:15:25:28] ENGINE Serving on https://172.21.15.90:7150 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:29 smithi090 ceph-mon[27811]: [13/Sep/2024:15:25:28] ENGINE Bus STARTED 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:29 smithi090 ceph-mon[27811]: [13/Sep/2024:15:25:28] ENGINE Error in HTTPServer.serve 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Traceback (most recent call last): 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._connections.run(self.expiration_interval) 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._run(expiration_interval) 2024-09-13T15:25:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: s = self.context.wrap_socket( 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: return self.sslsocket_class._create( 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self.do_handshake() 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._sslobj.do_handshake() 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T15:25:30.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:29 smithi090 ceph-mon[27811]: mgrmap e8: smithi090.fyvvpd(active, since 2s) 2024-09-13T15:25:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:33 smithi090 ceph-mon[27811]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-13T15:25:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:33 smithi090 ceph-mon[27811]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-13T15:25:38.633 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.810+0000 7f8e0c229640 1 Processor -- start 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.810+0000 7f8e0c229640 1 -- start start 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.810+0000 7f8e0c229640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.811+0000 7f8e0c229640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e041054e0 con 0x7f8e04104b10 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.811+0000 7f8e09fc5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.811+0000 7f8e09fc5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53104/0 (socket says 172.21.15.90:53104) 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.811+0000 7f8e09fc5640 1 -- 172.21.15.90:0/272336689 learned_addr learned my addr 172.21.15.90:0/272336689 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:38.634 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.811+0000 7f8e09fc5640 1 -- 172.21.15.90:0/272336689 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04105d10 con 0x7f8e04104b10 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e09fc5640 1 --2- 172.21.15.90:0/272336689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04104f10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009e80 tx=0x7f8e0002a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75cf51b4a4bebcc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e08fc3640 1 -- 172.21.15.90:0/272336689 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e00031400 con 0x7f8e04104b10 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e08fc3640 1 -- 172.21.15.90:0/272336689 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f8e0002d050 con 0x7f8e04104b10 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e08fc3640 1 -- 172.21.15.90:0/272336689 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e000077c0 con 0x7f8e04104b10 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e0c229640 1 -- 172.21.15.90:0/272336689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 msgr2=0x7f8e04104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e0c229640 1 --2- 172.21.15.90:0/272336689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04104f10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009e80 tx=0x7f8e0002a7a0 comp rx=0 tx=0).stop 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e0c229640 1 -- 172.21.15.90:0/272336689 shutdown_connections 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e0c229640 1 --2- 172.21.15.90:0/272336689 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04104f10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.812+0000 7f8e0c229640 1 -- 172.21.15.90:0/272336689 >> 172.21.15.90:0/272336689 conn(0x7f8e041002e0 msgr2=0x7f8e04102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e0c229640 1 -- 172.21.15.90:0/272336689 shutdown_connections 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e0c229640 1 -- 172.21.15.90:0/272336689 wait complete. 2024-09-13T15:25:38.635 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e0c229640 1 Processor -- start 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e0c229640 1 -- start start 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e0c229640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e0c229640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e04198db0 con 0x7f8e04104b10 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e09fc5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e09fc5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:53120/0 (socket says 172.21.15.90:53120) 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.813+0000 7f8e09fc5640 1 -- 172.21.15.90:0/2370267109 learned_addr learned my addr 172.21.15.90:0/2370267109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8e09fc5640 1 -- 172.21.15.90:0/2370267109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e00009b30 con 0x7f8e04104b10 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8e09fc5640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04198870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009fa0 tx=0x7f8e00031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e00031b80 con 0x7f8e04104b10 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f8e00031ce0 con 0x7f8e04104b10 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e00039620 con 0x7f8e04104b10 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e04106b90 con 0x7f8e04104b10 2024-09-13T15:25:38.636 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.814+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e041992c0 con 0x7f8e04104b10 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.815+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f8e00038040 con 0x7f8e04104b10 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.815+0000 7f8dfaffd640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f8df0038760 0x7f8df003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.815+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8df003b330 con 0x7f8df0038760 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.815+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8e0006b820 con 0x7f8e04104b10 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.815+0000 7f8e097c4640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f8df0038760 0x7f8df003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.816+0000 7f8e097c4640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f8df0038760 0x7f8df003ac20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8df40098c0 tx=0x7f8df4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.816+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f8df003b330 con 0x7f8df0038760 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.821+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8e04199500 con 0x7f8df0038760 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.821+0000 7f8dfaffd640 1 -- 172.21.15.90:0/2370267109 <== mgr.14118 v2:172.21.15.90:6800/3565908774 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f8e04199500 con 0x7f8df0038760 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f8df0038760 msgr2=0x7f8df003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f8df0038760 0x7f8df003ac20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8df40098c0 tx=0x7f8df4006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:38.637 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 msgr2=0x7f8e04198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04198870 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009fa0 tx=0x7f8e00031630 comp rx=0 tx=0).stop 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 shutdown_connections 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f8df0038760 0x7f8df003ac20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 --2- 172.21.15.90:0/2370267109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8e04104b10 0x7f8e04198870 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 >> 172.21.15.90:0/2370267109 conn(0x7f8e041002e0 msgr2=0x7f8e04100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 shutdown_connections 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:32.822+0000 7f8e0c229640 1 -- 172.21.15.90:0/2370267109 wait complete. 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 5 is available 2024-09-13T15:25:38.638 INFO:teuthology.orchestra.run.smithi090.stdout:Setting orchestrator backend to cephadm... 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2faa931640 1 Processor -- start 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2faa931640 1 -- start start 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2faa931640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa41037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2faa931640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa4077e30 con 0x7f2fa41033a0 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa41037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa41037a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54164/0 (socket says 172.21.15.90:54164) 2024-09-13T15:25:39.765 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2fa3fff640 1 -- 172.21.15.90:0/2526220346 learned_addr learned my addr 172.21.15.90:0/2526220346 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.009+0000 7f2fa3fff640 1 -- 172.21.15.90:0/2526220346 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa4103ce0 con 0x7f2fa41033a0 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.010+0000 7f2fa3fff640 1 --2- 172.21.15.90:0/2526220346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa41037a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009820 tx=0x7f2f9402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542bbf613795939f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.010+0000 7f2fa2ffd640 1 -- 172.21.15.90:0/2526220346 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9402ad40 con 0x7f2fa41033a0 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.010+0000 7f2fa2ffd640 1 -- 172.21.15.90:0/2526220346 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2f940328b0 con 0x7f2fa41033a0 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.010+0000 7f2fa2ffd640 1 -- 172.21.15.90:0/2526220346 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f940307a0 con 0x7f2fa41033a0 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 -- 172.21.15.90:0/2526220346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 msgr2=0x7f2fa41037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 --2- 172.21.15.90:0/2526220346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa41037a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009820 tx=0x7f2f9402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 -- 172.21.15.90:0/2526220346 shutdown_connections 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 --2- 172.21.15.90:0/2526220346 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa41037a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 -- 172.21.15.90:0/2526220346 >> 172.21.15.90:0/2526220346 conn(0x7f2fa40ff1b0 msgr2=0x7f2fa41015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 -- 172.21.15.90:0/2526220346 shutdown_connections 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.011+0000 7f2faa931640 1 -- 172.21.15.90:0/2526220346 wait complete. 2024-09-13T15:25:39.766 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2faa931640 1 Processor -- start 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2faa931640 1 -- start start 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2faa931640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2faa931640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa419ade0 con 0x7f2fa41033a0 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2fa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54176/0 (socket says 172.21.15.90:54176) 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2fa3fff640 1 -- 172.21.15.90:0/2594508904 learned_addr learned my addr 172.21.15.90:0/2594508904 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.012+0000 7f2fa3fff640 1 -- 172.21.15.90:0/2594508904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940094d0 con 0x7f2fa41033a0 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.013+0000 7f2fa3fff640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa419a8a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2f94030490 tx=0x7f2f94032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.013+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f94030eb0 con 0x7f2fa41033a0 2024-09-13T15:25:39.767 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.013+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2f94032c30 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.013+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa419afe0 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.013+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f9403a560 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.013+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa419b480 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.014+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f2f9403a820 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.014+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa4104cb0 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.014+0000 7f2fa17fa640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f2f8c038710 0x7f2f8c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.014+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2f9406bb70 con 0x7f2fa41033a0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.014+0000 7f2fa37fe640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f2f8c038710 0x7f2f8c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:39.768 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.015+0000 7f2fa37fe640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f2f8c038710 0x7f2f8c03abd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f980098c0 tx=0x7f2f98006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.019+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f2f94043310 con 0x7f2fa41033a0 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.193+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fa419ddc0 con 0x7f2f8c038710 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.200+0000 7f2fa17fa640 1 -- 172.21.15.90:0/2594508904 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2fa419ddc0 con 0x7f2f8c038710 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f2f8c038710 msgr2=0x7f2f8c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f2f8c038710 0x7f2f8c03abd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f980098c0 tx=0x7f2f98006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 msgr2=0x7f2fa419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa419a8a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2f94030490 tx=0x7f2f94032840 comp rx=0 tx=0).stop 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 shutdown_connections 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f2f8c038710 0x7f2f8c03abd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 --2- 172.21.15.90:0/2594508904 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2fa41033a0 0x7f2fa419a8a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 >> 172.21.15.90:0/2594508904 conn(0x7f2fa40ff1b0 msgr2=0x7f2fa418efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:39.769 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 shutdown_connections 2024-09-13T15:25:39.770 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:39.202+0000 7f2faa931640 1 -- 172.21.15.90:0/2594508904 wait complete. 2024-09-13T15:25:40.303 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:40 smithi090 ceph-mon[27811]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:40.303 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:40 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:40.303 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:40 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:25:40.847 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f726dfe9640 1 Processor -- start 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f726dfe9640 1 -- start start 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f726dfe9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681031b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f726dfe9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268103780 con 0x7f7268102db0 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f72677fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f72677fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54188/0 (socket says 172.21.15.90:54188) 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f72677fe640 1 -- 172.21.15.90:0/2693138296 learned_addr learned my addr 172.21.15.90:0/2693138296 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.127+0000 7f72677fe640 1 -- 172.21.15.90:0/2693138296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72681038c0 con 0x7f7268102db0 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.128+0000 7f72677fe640 1 --2- 172.21.15.90:0/2693138296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681031b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7258009820 tx=0x7f725802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2597ea5f9add20c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:40.848 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.128+0000 7f72667fc640 1 -- 172.21.15.90:0/2693138296 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f725802ad40 con 0x7f7268102db0 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.128+0000 7f72667fc640 1 -- 172.21.15.90:0/2693138296 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f72580328b0 con 0x7f7268102db0 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.128+0000 7f72667fc640 1 -- 172.21.15.90:0/2693138296 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72580307a0 con 0x7f7268102db0 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 -- 172.21.15.90:0/2693138296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 msgr2=0x7f72681031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 --2- 172.21.15.90:0/2693138296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681031b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7258009820 tx=0x7f725802a270 comp rx=0 tx=0).stop 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 -- 172.21.15.90:0/2693138296 shutdown_connections 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 --2- 172.21.15.90:0/2693138296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681031b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 -- 172.21.15.90:0/2693138296 >> 172.21.15.90:0/2693138296 conn(0x7f72680fe960 msgr2=0x7f7268100d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 -- 172.21.15.90:0/2693138296 shutdown_connections 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 -- 172.21.15.90:0/2693138296 wait complete. 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.129+0000 7f726dfe9640 1 Processor -- start 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f726dfe9640 1 -- start start 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f726dfe9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:40.849 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f726dfe9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7268198c30 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f72677fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f72677fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54196/0 (socket says 172.21.15.90:54196) 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f72677fe640 1 -- 172.21.15.90:0/2762497744 learned_addr learned my addr 172.21.15.90:0/2762497744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f72677fe640 1 -- 172.21.15.90:0/2762497744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580094d0 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f72677fe640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681986f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7258030490 tx=0x7f7258032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7258030e80 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f7258032c30 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.130+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7268198e30 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.131+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f725803a560 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.131+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72681992d0 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.131+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f725803c400 con 0x7f7268102db0 2024-09-13T15:25:40.850 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.132+0000 7f7264ff9640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7250038760 0x7f725003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.132+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7268106a80 con 0x7f7268102db0 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.132+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f725806b960 con 0x7f7268102db0 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.132+0000 7f7266ffd640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7250038760 0x7f725003ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.132+0000 7f7266ffd640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7250038760 0x7f725003ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f725c0098c0 tx=0x7f725c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.136+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f72580319f0 con 0x7f7268102db0 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.298+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f72680ffbf0 con 0x7f7250038760 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.299+0000 7f7264ff9640 1 -- 172.21.15.90:0/2762497744 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f72680ffbf0 con 0x7f7250038760 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.301+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7250038760 msgr2=0x7f725003ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.301+0000 7f726dfe9640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7250038760 0x7f725003ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f725c0098c0 tx=0x7f725c006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.301+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 msgr2=0x7f72681986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.301+0000 7f726dfe9640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681986f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7258030490 tx=0x7f7258032840 comp rx=0 tx=0).stop 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.302+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 shutdown_connections 2024-09-13T15:25:40.851 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.302+0000 7f726dfe9640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7250038760 0x7f725003ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:40.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.302+0000 7f726dfe9640 1 --2- 172.21.15.90:0/2762497744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7268102db0 0x7f72681986f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:40.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.302+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 >> 172.21.15.90:0/2762497744 conn(0x7f72680fe960 msgr2=0x7f72680ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:40.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.302+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 shutdown_connections 2024-09-13T15:25:40.852 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:40.302+0000 7f726dfe9640 1 -- 172.21.15.90:0/2762497744 wait complete. 2024-09-13T15:25:40.852 INFO:teuthology.orchestra.run.smithi090.stdout:Generating ssh key... 2024-09-13T15:25:41.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:41 smithi090 ceph-mon[27811]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4983ec3640 1 Processor -- start 2024-09-13T15:25:42.207 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4983ec3640 1 -- start start 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4983ec3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c107980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4983ec3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c077e50 con 0x7f497c105590 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4981c5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c107980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4981c5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c107980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54204/0 (socket says 172.21.15.90:54204) 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4981c5f640 1 -- 172.21.15.90:0/2632744937 learned_addr learned my addr 172.21.15.90:0/2632744937 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.201+0000 7f4981c5f640 1 -- 172.21.15.90:0/2632744937 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c107ec0 con 0x7f497c105590 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.202+0000 7f4981c5f640 1 --2- 172.21.15.90:0/2632744937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c107980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4978009820 tx=0x7f497802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526eb335f9b5cac3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.202+0000 7f4980c5d640 1 -- 172.21.15.90:0/2632744937 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f497802ad40 con 0x7f497c105590 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.202+0000 7f4980c5d640 1 -- 172.21.15.90:0/2632744937 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f49780328b0 con 0x7f497c105590 2024-09-13T15:25:42.208 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.202+0000 7f4980c5d640 1 -- 172.21.15.90:0/2632744937 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49780307a0 con 0x7f497c105590 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 -- 172.21.15.90:0/2632744937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 msgr2=0x7f497c107980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 --2- 172.21.15.90:0/2632744937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c107980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4978009820 tx=0x7f497802a270 comp rx=0 tx=0).stop 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 -- 172.21.15.90:0/2632744937 shutdown_connections 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 --2- 172.21.15.90:0/2632744937 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c107980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 -- 172.21.15.90:0/2632744937 >> 172.21.15.90:0/2632744937 conn(0x7f497c0ff110 msgr2=0x7f497c101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 -- 172.21.15.90:0/2632744937 shutdown_connections 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.203+0000 7f4983ec3640 1 -- 172.21.15.90:0/2632744937 wait complete. 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4983ec3640 1 Processor -- start 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4983ec3640 1 -- start start 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4983ec3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:42.209 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4983ec3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c198c50 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4981c5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4981c5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54214/0 (socket says 172.21.15.90:54214) 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4981c5f640 1 -- 172.21.15.90:0/3963386218 learned_addr learned my addr 172.21.15.90:0/3963386218 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4981c5f640 1 -- 172.21.15.90:0/3963386218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780094d0 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4981c5f640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c198710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49780097f0 tx=0x7f4978030c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4978040070 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c198e50 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4978032c30 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f497803a560 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.204+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c1992f0 con 0x7f497c105590 2024-09-13T15:25:42.210 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.205+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f497803a790 con 0x7f497c105590 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.205+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c108c20 con 0x7f497c105590 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.205+0000 7f4972ffd640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f4968038710 0x7f496803abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.206+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f497806b740 con 0x7f497c105590 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.206+0000 7f498145e640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f4968038710 0x7f496803abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.206+0000 7f498145e640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f4968038710 0x7f496803abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098c0 tx=0x7f496c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.210+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f4978045740 con 0x7f497c105590 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.370+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f497c100450 con 0x7f4968038710 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.630+0000 7f4972ffd640 1 -- 172.21.15.90:0/3963386218 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f497c100450 con 0x7f4968038710 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f4968038710 msgr2=0x7f496803abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f4968038710 0x7f496803abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098c0 tx=0x7f496c006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:42.211 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 msgr2=0x7f497c198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c198710 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f49780097f0 tx=0x7f4978030c90 comp rx=0 tx=0).stop 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 shutdown_connections 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f4968038710 0x7f496803abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 --2- 172.21.15.90:0/3963386218 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f497c105590 0x7f497c198710 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.632+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 >> 172.21.15.90:0/3963386218 conn(0x7f497c0ff110 msgr2=0x7f497c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.633+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 shutdown_connections 2024-09-13T15:25:42.212 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:41.633+0000 7f4983ec3640 1 -- 172.21.15.90:0/3963386218 wait complete. 2024-09-13T15:25:42.745 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:42 smithi090 ceph-mon[27811]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:42.745 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:42 smithi090 ceph-mon[27811]: Generating ssh key... 2024-09-13T15:25:42.745 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:42 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:42.745 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:42 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:43.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqIQdihJcO89NCks6rs+mLqYXSGlM24HqpneJm5+KqQv66tEGe/2YujLyHNTuXLd9nAgi3ufYAikcmdFOXAilUQYwwQ70o7XbKgFQsLt3iCT5S9uoBO6Y0WhNan3Jvlteqw8/vmZwuyDwOU2iQ5YgTkU+yRywhD/OIkXAqltYOMvOCdRoUbuch129NjpuWLbGKaA7nKOztpEMEY2Odtx39Nvk0ii3qivABnI2CpDbKE1Hngje3sfng7etewm4cOSBFDDvWuhk9ny6zoF6w7ah9UHi9KPUV2q89ioYwOEsbb+/riJK51RZRkqPgYgS1A4jV48Pvwz0znM6Gjh7TJ/FNvocmxgrZ6/k0MnbTxJ8k8RX/qDpkDJCOoCcJrRDH7qNbvQAHDpyD87SUcMx05k6muIZBmn9WNXSKwPtc1SpnRzSYxUSBqKPf1vvhY4Hx+6LEz6naBjs15sWMSNTRhzG+YL6Vd6N8Sq5gDIHBYgSOdBE+lsSPxRvQ3uRnt/pHSUk= ceph-41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.569+0000 7f86be306640 1 Processor -- start 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.569+0000 7f86be306640 1 -- start start 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.569+0000 7f86be306640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.569+0000 7f86be306640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b81054e0 con 0x7f86b8104b10 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.570+0000 7f86b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.570+0000 7f86b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54224/0 (socket says 172.21.15.90:54224) 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.570+0000 7f86b7fff640 1 -- 172.21.15.90:0/4198063327 learned_addr learned my addr 172.21.15.90:0/4198063327 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.570+0000 7f86b7fff640 1 -- 172.21.15.90:0/4198063327 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8105d10 con 0x7f86b8104b10 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.570+0000 7f86b7fff640 1 --2- 172.21.15.90:0/4198063327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8104f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009820 tx=0x7f86a802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e63da28e1817245 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:43.300 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86b6ffd640 1 -- 172.21.15.90:0/4198063327 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a802ad40 con 0x7f86b8104b10 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86b6ffd640 1 -- 172.21.15.90:0/4198063327 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f86a80328b0 con 0x7f86b8104b10 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86b6ffd640 1 -- 172.21.15.90:0/4198063327 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a80307a0 con 0x7f86b8104b10 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 -- 172.21.15.90:0/4198063327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 msgr2=0x7f86b8104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 --2- 172.21.15.90:0/4198063327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8104f10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009820 tx=0x7f86a802a270 comp rx=0 tx=0).stop 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 -- 172.21.15.90:0/4198063327 shutdown_connections 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 --2- 172.21.15.90:0/4198063327 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8104f10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 -- 172.21.15.90:0/4198063327 >> 172.21.15.90:0/4198063327 conn(0x7f86b81002e0 msgr2=0x7f86b8102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 -- 172.21.15.90:0/4198063327 shutdown_connections 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.571+0000 7f86be306640 1 -- 172.21.15.90:0/4198063327 wait complete. 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86be306640 1 Processor -- start 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86be306640 1 -- start start 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86be306640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:43.301 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86be306640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b8198db0 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86b7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54226/0 (socket says 172.21.15.90:54226) 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.572+0000 7f86b7fff640 1 -- 172.21.15.90:0/3802263350 learned_addr learned my addr 172.21.15.90:0/3802263350 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86b7fff640 1 -- 172.21.15.90:0/3802263350 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a80094d0 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86b7fff640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8198870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f86a80097f0 tx=0x7f86a8030d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a8040070 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b8106b90 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f86a8032c30 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86a803a560 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.573+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b81992c0 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.574+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f86a803c400 con 0x7f86b8104b10 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.574+0000 7f86b57fa640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f86a0038710 0x7f86a003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:43.302 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.574+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b8199500 con 0x7f86b8104b10 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.574+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f86a806b5e0 con 0x7f86b8104b10 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.574+0000 7f86b77fe640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f86a0038710 0x7f86a003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.574+0000 7f86b77fe640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f86a0038710 0x7f86a003abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0098c0 tx=0x7f86ac006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.579+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f86a803c900 con 0x7f86b8104b10 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.743+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f86b81014a0 con 0x7f86a0038710 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.743+0000 7f86b57fa640 1 -- 172.21.15.90:0/3802263350 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f86b81014a0 con 0x7f86a0038710 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f86a0038710 msgr2=0x7f86a003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f86a0038710 0x7f86a003abd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0098c0 tx=0x7f86ac006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 msgr2=0x7f86b8198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8198870 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f86a80097f0 tx=0x7f86a8030d60 comp rx=0 tx=0).stop 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 shutdown_connections 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f86a0038710 0x7f86a003abd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:43.303 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 --2- 172.21.15.90:0/3802263350 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f86b8104b10 0x7f86b8198870 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:43.304 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 >> 172.21.15.90:0/3802263350 conn(0x7f86b81002e0 msgr2=0x7f86b8100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:43.304 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 shutdown_connections 2024-09-13T15:25:43.304 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:42.746+0000 7f86be306640 1 -- 172.21.15.90:0/3802263350 wait complete. 2024-09-13T15:25:43.304 INFO:teuthology.orchestra.run.smithi090.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:25:43.304 INFO:teuthology.orchestra.run.smithi090.stdout:Adding key to root@localhost authorized_keys... 2024-09-13T15:25:43.304 INFO:teuthology.orchestra.run.smithi090.stdout:Adding host smithi090... 2024-09-13T15:25:43.673 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:43 smithi090 ceph-mon[27811]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:44.874 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:44 smithi090 ceph-mon[27811]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:45.809 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:45 smithi090 ceph-mon[27811]: Deploying cephadm binary to smithi090 2024-09-13T15:25:47.243 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Added host 'smithi090' with addr '172.21.15.90' 2024-09-13T15:25:47.243 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.671+0000 7fd5908df640 1 Processor -- start 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.671+0000 7fd5908df640 1 -- start start 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.672+0000 7fd5908df640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c103190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.672+0000 7fd5908df640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c103760 con 0x7fd58c102d90 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.672+0000 7fd58a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.672+0000 7fd58a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54228/0 (socket says 172.21.15.90:54228) 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.672+0000 7fd58a59c640 1 -- 172.21.15.90:0/3345222109 learned_addr learned my addr 172.21.15.90:0/3345222109 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.672+0000 7fd58a59c640 1 -- 172.21.15.90:0/3345222109 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c1038a0 con 0x7fd58c102d90 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.673+0000 7fd58a59c640 1 --2- 172.21.15.90:0/3345222109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c103190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009820 tx=0x7fd57c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00ffed9940bb928 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.673+0000 7fd58959a640 1 -- 172.21.15.90:0/3345222109 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd57c02ad40 con 0x7fd58c102d90 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.673+0000 7fd58959a640 1 -- 172.21.15.90:0/3345222109 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd57c0328b0 con 0x7fd58c102d90 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.673+0000 7fd58959a640 1 -- 172.21.15.90:0/3345222109 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd57c0307a0 con 0x7fd58c102d90 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 -- 172.21.15.90:0/3345222109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 msgr2=0x7fd58c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 --2- 172.21.15.90:0/3345222109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c103190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009820 tx=0x7fd57c02a270 comp rx=0 tx=0).stop 2024-09-13T15:25:47.244 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 -- 172.21.15.90:0/3345222109 shutdown_connections 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 --2- 172.21.15.90:0/3345222109 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c103190 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 -- 172.21.15.90:0/3345222109 >> 172.21.15.90:0/3345222109 conn(0x7fd58c0fe8e0 msgr2=0x7fd58c100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 -- 172.21.15.90:0/3345222109 shutdown_connections 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 -- 172.21.15.90:0/3345222109 wait complete. 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.674+0000 7fd5908df640 1 Processor -- start 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd5908df640 1 -- start start 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd5908df640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd5908df640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58c19add0 con 0x7fd58c102d90 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd58a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd58a59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c19a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:54232/0 (socket says 172.21.15.90:54232) 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd58a59c640 1 -- 172.21.15.90:0/2567652367 learned_addr learned my addr 172.21.15.90:0/2567652367 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd58a59c640 1 -- 172.21.15.90:0/2567652367 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c0094d0 con 0x7fd58c102d90 2024-09-13T15:25:47.245 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd58a59c640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c19a890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0097f0 tx=0x7fd57c030dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd57c032c30 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd57c032d90 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd58c19afd0 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd57c03a560 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.675+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd58c19b470 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.676+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fd57c03a790 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.677+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd58c1047e0 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.677+0000 7fd57b7fe640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fd574038760 0x7fd57403ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.677+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd57c06b760 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.677+0000 7fd589d9b640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fd574038760 0x7fd57403ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.677+0000 7fd589d9b640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fd574038760 0x7fd57403ac20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd5800098c0 tx=0x7fd580006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.681+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd57c039960 con 0x7fd58c102d90 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:43.847+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi090", "addr": "172.21.15.90", "target": ["mon-mgr", ""]}) v1 -- 0x7fd58c0608f0 con 0x7fd574038760 2024-09-13T15:25:47.246 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.683+0000 7fd57b7fe640 1 -- 172.21.15.90:0/2567652367 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fd58c0608f0 con 0x7fd574038760 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.685+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fd574038760 msgr2=0x7fd57403ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.685+0000 7fd5908df640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fd574038760 0x7fd57403ac20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd5800098c0 tx=0x7fd580006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.685+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 msgr2=0x7fd58c19a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.685+0000 7fd5908df640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c19a890 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0097f0 tx=0x7fd57c030dd0 comp rx=0 tx=0).stop 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.685+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 shutdown_connections 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.686+0000 7fd5908df640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fd574038760 0x7fd57403ac20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.686+0000 7fd5908df640 1 --2- 172.21.15.90:0/2567652367 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd58c102d90 0x7fd58c19a890 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.686+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 >> 172.21.15.90:0/2567652367 conn(0x7fd58c0fe8e0 msgr2=0x7fd58c191080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.686+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 shutdown_connections 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:46.686+0000 7fd5908df640 1 -- 172.21.15.90:0/2567652367 wait complete. 2024-09-13T15:25:47.247 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying mon service with default placement... 2024-09-13T15:25:47.939 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:47 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:47.939 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:47 smithi090 ceph-mon[27811]: Added host smithi090 2024-09-13T15:25:47.939 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:47 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:25:47.939 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:47 smithi090 ceph-mon[27811]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:25:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:48 smithi090 ceph-mon[27811]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:48 smithi090 ceph-mon[27811]: Saving service mon spec with placement count:5 2024-09-13T15:25:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:48 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:49.293 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-13T15:25:49.293 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede9a81640 1 Processor -- start 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede9a81640 1 -- start start 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede9a81640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede9a81640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede41054e0 con 0x7fede4104b10 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56740/0 (socket says 172.21.15.90:56740) 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.633+0000 7fede37fe640 1 -- 172.21.15.90:0/2661596022 learned_addr learned my addr 172.21.15.90:0/2661596022 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.634+0000 7fede37fe640 1 -- 172.21.15.90:0/2661596022 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede4105d10 con 0x7fede4104b10 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.634+0000 7fede37fe640 1 --2- 172.21.15.90:0/2661596022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4104f10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009820 tx=0x7fedd402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef9056841c11faa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.634+0000 7fede27fc640 1 -- 172.21.15.90:0/2661596022 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedd402ad40 con 0x7fede4104b10 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede27fc640 1 -- 172.21.15.90:0/2661596022 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fedd40328b0 con 0x7fede4104b10 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede27fc640 1 -- 172.21.15.90:0/2661596022 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedd40307a0 con 0x7fede4104b10 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 -- 172.21.15.90:0/2661596022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 msgr2=0x7fede4104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:49.294 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 --2- 172.21.15.90:0/2661596022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4104f10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009820 tx=0x7fedd402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 -- 172.21.15.90:0/2661596022 shutdown_connections 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 --2- 172.21.15.90:0/2661596022 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4104f10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 -- 172.21.15.90:0/2661596022 >> 172.21.15.90:0/2661596022 conn(0x7fede41002e0 msgr2=0x7fede4102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 -- 172.21.15.90:0/2661596022 shutdown_connections 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.635+0000 7fede9a81640 1 -- 172.21.15.90:0/2661596022 wait complete. 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede9a81640 1 Processor -- start 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede9a81640 1 -- start start 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede9a81640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede9a81640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede4198db0 con 0x7fede4104b10 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56754/0 (socket says 172.21.15.90:56754) 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede37fe640 1 -- 172.21.15.90:0/2577572070 learned_addr learned my addr 172.21.15.90:0/2577572070 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.636+0000 7fede37fe640 1 -- 172.21.15.90:0/2577572070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40094d0 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.637+0000 7fede37fe640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4198870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fedd4030490 tx=0x7fedd4032840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.637+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedd4030e80 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.637+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fedd4032c30 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.637+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedd403a560 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.637+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede4106b90 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.637+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede41992c0 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.638+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fedd403a7d0 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.638+0000 7fede0ff9640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedcc038710 0x7fedcc03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.638+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fedd406b990 con 0x7fede4104b10 2024-09-13T15:25:49.296 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.638+0000 7fede2ffd640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedcc038710 0x7fedcc03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.638+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede4199500 con 0x7fede4104b10 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.638+0000 7fede2ffd640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedcc038710 0x7fedcc03abd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098c0 tx=0x7fedd8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.643+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fedd4046460 con 0x7fede4104b10 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.840+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fede41014a0 con 0x7fedcc038710 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.847+0000 7fede0ff9640 1 -- 172.21.15.90:0/2577572070 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fede41014a0 con 0x7fedcc038710 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedcc038710 msgr2=0x7fedcc03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedcc038710 0x7fedcc03abd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fedd80098c0 tx=0x7fedd8006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 msgr2=0x7fede4198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4198870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fedd4030490 tx=0x7fedd4032840 comp rx=0 tx=0).stop 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 shutdown_connections 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedcc038710 0x7fedcc03abd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 --2- 172.21.15.90:0/2577572070 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fede4104b10 0x7fede4198870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:49.297 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.849+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 >> 172.21.15.90:0/2577572070 conn(0x7fede41002e0 msgr2=0x7fede4100d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:49.298 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.850+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 shutdown_connections 2024-09-13T15:25:49.298 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:47.850+0000 7fede9a81640 1 -- 172.21.15.90:0/2577572070 wait complete. 2024-09-13T15:25:49.298 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying mgr service with default placement... 2024-09-13T15:25:50.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:49 smithi090 ceph-mon[27811]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:25:50.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:49 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:50.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:49 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:50.436 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-13T15:25:50.437 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.681+0000 7f5f917df640 1 Processor -- start 2024-09-13T15:25:50.437 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f917df640 1 -- start start 2024-09-13T15:25:50.437 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f917df640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f917df640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c1054e0 con 0x7f5f8c104b10 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f8affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f8affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56756/0 (socket says 172.21.15.90:56756) 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f8affd640 1 -- 172.21.15.90:0/2456528944 learned_addr learned my addr 172.21.15.90:0/2456528944 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.682+0000 7f5f8affd640 1 -- 172.21.15.90:0/2456528944 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c105d10 con 0x7f5f8c104b10 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.683+0000 7f5f8affd640 1 --2- 172.21.15.90:0/2456528944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c104f10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009e80 tx=0x7f5f7c02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fc5c8f5c5191db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.683+0000 7f5f89ffb640 1 -- 172.21.15.90:0/2456528944 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7c031400 con 0x7f5f8c104b10 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.683+0000 7f5f89ffb640 1 -- 172.21.15.90:0/2456528944 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5f7c02d050 con 0x7f5f8c104b10 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.683+0000 7f5f89ffb640 1 -- 172.21.15.90:0/2456528944 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7c007720 con 0x7f5f8c104b10 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.683+0000 7f5f917df640 1 -- 172.21.15.90:0/2456528944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 msgr2=0x7f5f8c104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.683+0000 7f5f917df640 1 --2- 172.21.15.90:0/2456528944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c104f10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009e80 tx=0x7f5f7c02a7a0 comp rx=0 tx=0).stop 2024-09-13T15:25:50.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 -- 172.21.15.90:0/2456528944 shutdown_connections 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 --2- 172.21.15.90:0/2456528944 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c104f10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 -- 172.21.15.90:0/2456528944 >> 172.21.15.90:0/2456528944 conn(0x7f5f8c1002e0 msgr2=0x7f5f8c102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 -- 172.21.15.90:0/2456528944 shutdown_connections 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 -- 172.21.15.90:0/2456528944 wait complete. 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 Processor -- start 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.684+0000 7f5f917df640 1 -- start start 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f917df640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f917df640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c07a0b0 con 0x7f5f8c104b10 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f8affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f8affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c079b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56760/0 (socket says 172.21.15.90:56760) 2024-09-13T15:25:50.439 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f8affd640 1 -- 172.21.15.90:0/4130271377 learned_addr learned my addr 172.21.15.90:0/4130271377 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f8affd640 1 -- 172.21.15.90:0/4130271377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c009b30 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f8affd640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c079b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009e50 tx=0x7f5f7c031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.685+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7c031ba0 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.686+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5f7c031d00 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.686+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f7c039550 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.686+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c0782a0 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.686+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c078740 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.687+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f5f7c038040 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.687+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c104f90 con 0x7f5f8c104b10 2024-09-13T15:25:50.440 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.687+0000 7f5f83fff640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f5f78038760 0x7f5f7803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.687+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5f7c06ba80 con 0x7f5f8c104b10 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.687+0000 7f5f8a7fc640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f5f78038760 0x7f5f7803ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.688+0000 7f5f8a7fc640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f5f78038760 0x7f5f7803ac20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5f740098c0 tx=0x7f5f74006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.692+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f5f7c04d300 con 0x7f5f8c104b10 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.877+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f8c078da0 con 0x7f5f78038760 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.885+0000 7f5f83fff640 1 -- 172.21.15.90:0/4130271377 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5f8c078da0 con 0x7f5f78038760 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f5f78038760 msgr2=0x7f5f7803ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f5f78038760 0x7f5f7803ac20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5f740098c0 tx=0x7f5f74006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 msgr2=0x7f5f8c079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c079b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009e50 tx=0x7f5f7c031630 comp rx=0 tx=0).stop 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 shutdown_connections 2024-09-13T15:25:50.441 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f5f78038760 0x7f5f7803ac20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:50.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 --2- 172.21.15.90:0/4130271377 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f8c104b10 0x7f5f8c079b70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:50.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 >> 172.21.15.90:0/4130271377 conn(0x7f5f8c1002e0 msgr2=0x7f5f8c100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:50.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 shutdown_connections 2024-09-13T15:25:50.442 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:49.887+0000 7f5f917df640 1 -- 172.21.15.90:0/4130271377 wait complete. 2024-09-13T15:25:50.442 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying crash service with default placement... 2024-09-13T15:25:51.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:50 smithi090 ceph-mon[27811]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:51.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:50 smithi090 ceph-mon[27811]: Saving service mgr spec with placement count:2 2024-09-13T15:25:51.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:50 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:51.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-09-13T15:25:51.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.848+0000 7ffb98baf640 1 Processor -- start 2024-09-13T15:25:51.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb98baf640 1 -- start start 2024-09-13T15:25:51.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb98baf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb941037e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:51.747 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb98baf640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb94077e50 con 0x7ffb941033e0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb9259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb941037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb9259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb941037e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56770/0 (socket says 172.21.15.90:56770) 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb9259c640 1 -- 172.21.15.90:0/1106245744 learned_addr learned my addr 172.21.15.90:0/1106245744 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.849+0000 7ffb9259c640 1 -- 172.21.15.90:0/1106245744 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94103d20 con 0x7ffb941033e0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.850+0000 7ffb9259c640 1 --2- 172.21.15.90:0/1106245744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb941037e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009820 tx=0x7ffb8402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83817c1845f0795f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.850+0000 7ffb9159a640 1 -- 172.21.15.90:0/1106245744 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb8402ad40 con 0x7ffb941033e0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.850+0000 7ffb9159a640 1 -- 172.21.15.90:0/1106245744 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffb840328d0 con 0x7ffb941033e0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.850+0000 7ffb9159a640 1 -- 172.21.15.90:0/1106245744 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb840306f0 con 0x7ffb941033e0 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 -- 172.21.15.90:0/1106245744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 msgr2=0x7ffb941037e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 --2- 172.21.15.90:0/1106245744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb941037e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009820 tx=0x7ffb8402a270 comp rx=0 tx=0).stop 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 -- 172.21.15.90:0/1106245744 shutdown_connections 2024-09-13T15:25:51.748 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 --2- 172.21.15.90:0/1106245744 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb941037e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 -- 172.21.15.90:0/1106245744 >> 172.21.15.90:0/1106245744 conn(0x7ffb940ff230 msgr2=0x7ffb94101650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 -- 172.21.15.90:0/1106245744 shutdown_connections 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.851+0000 7ffb98baf640 1 -- 172.21.15.90:0/1106245744 wait complete. 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb98baf640 1 Processor -- start 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb98baf640 1 -- start start 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb98baf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb9419a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb98baf640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9419ae10 con 0x7ffb941033e0 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb9259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb9419a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb9259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb9419a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56774/0 (socket says 172.21.15.90:56774) 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb9259c640 1 -- 172.21.15.90:0/46291913 learned_addr learned my addr 172.21.15.90:0/46291913 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb9259c640 1 -- 172.21.15.90:0/46291913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb840094d0 con 0x7ffb941033e0 2024-09-13T15:25:51.749 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.852+0000 7ffb9259c640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb9419a8d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097f0 tx=0x7ffb84032710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.853+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb84030c80 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.853+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9419b010 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.853+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffb84030de0 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.853+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb8403a560 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.853+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9419b4b0 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.853+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7ffb8403a7e0 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.854+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb94104d30 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.854+0000 7ffb8b7fe640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ffb80038710 0x7ffb8003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.854+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffb8406b940 con 0x7ffb941033e0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.854+0000 7ffb91d9b640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ffb80038710 0x7ffb8003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.854+0000 7ffb91d9b640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ffb80038710 0x7ffb8003abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0098c0 tx=0x7ffb7c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:51.750 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:50.858+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7ffb84039d50 con 0x7ffb941033e0 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.054+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb940608f0 con 0x7ffb80038710 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.058+0000 7ffb8b7fe640 1 -- 172.21.15.90:0/46291913 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ffb940608f0 con 0x7ffb80038710 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ffb80038710 msgr2=0x7ffb8003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ffb80038710 0x7ffb8003abd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c0098c0 tx=0x7ffb7c006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 msgr2=0x7ffb9419a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb9419a8d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffb840097f0 tx=0x7ffb84032710 comp rx=0 tx=0).stop 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 shutdown_connections 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ffb80038710 0x7ffb8003abd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.060+0000 7ffb98baf640 1 --2- 172.21.15.90:0/46291913 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb941033e0 0x7ffb9419a8d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.061+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 >> 172.21.15.90:0/46291913 conn(0x7ffb940ff230 msgr2=0x7ffb9418f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.061+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 shutdown_connections 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:51.061+0000 7ffb98baf640 1 -- 172.21.15.90:0/46291913 wait complete. 2024-09-13T15:25:51.751 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying ceph-exporter service with default placement... 2024-09-13T15:25:51.989 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:51 smithi090 ceph-mon[27811]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:25:51.989 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:51 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:53.000 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled ceph-exporter update... 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc5a4936640 1 Processor -- start 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc5a4936640 1 -- start start 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc5a4936640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a0078340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc5a4936640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0078910 con 0x7fc5a0079ee0 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a0078340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a0078340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56790/0 (socket says 172.21.15.90:56790) 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.142+0000 7fc59e59c640 1 -- 172.21.15.90:0/2348815352 learned_addr learned my addr 172.21.15.90:0/2348815352 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:53.001 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.143+0000 7fc59e59c640 1 -- 172.21.15.90:0/2348815352 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a0078a50 con 0x7fc5a0079ee0 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.143+0000 7fc59e59c640 1 --2- 172.21.15.90:0/2348815352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a0078340 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc590009820 tx=0x7fc59002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2d12b6b59e803c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.143+0000 7fc59d59a640 1 -- 172.21.15.90:0/2348815352 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59002ad40 con 0x7fc5a0079ee0 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.143+0000 7fc59d59a640 1 -- 172.21.15.90:0/2348815352 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5900328d0 con 0x7fc5a0079ee0 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.143+0000 7fc59d59a640 1 -- 172.21.15.90:0/2348815352 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5900306f0 con 0x7fc5a0079ee0 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 -- 172.21.15.90:0/2348815352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 msgr2=0x7fc5a0078340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 --2- 172.21.15.90:0/2348815352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a0078340 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc590009820 tx=0x7fc59002a270 comp rx=0 tx=0).stop 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 -- 172.21.15.90:0/2348815352 shutdown_connections 2024-09-13T15:25:53.002 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 --2- 172.21.15.90:0/2348815352 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a0078340 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 -- 172.21.15.90:0/2348815352 >> 172.21.15.90:0/2348815352 conn(0x7fc5a0100260 msgr2=0x7fc5a01026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 -- 172.21.15.90:0/2348815352 shutdown_connections 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.144+0000 7fc5a4936640 1 -- 172.21.15.90:0/2348815352 wait complete. 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc5a4936640 1 Processor -- start 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc5a4936640 1 -- start start 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc5a4936640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc5a4936640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a019ae90 con 0x7fc5a0079ee0 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56794/0 (socket says 172.21.15.90:56794) 2024-09-13T15:25:53.003 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc59e59c640 1 -- 172.21.15.90:0/3200868515 learned_addr learned my addr 172.21.15.90:0/3200868515 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc59e59c640 1 -- 172.21.15.90:0/3200868515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5900094d0 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.145+0000 7fc59e59c640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a019a950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc590009940 tx=0x7fc590032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.146+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590030c10 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.146+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a019b090 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.146+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc590030d70 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.146+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59003a560 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.146+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a019b530 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.146+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fc59003a820 con 0x7fc5a0079ee0 2024-09-13T15:25:53.004 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.147+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a010aee0 con 0x7fc5a0079ee0 2024-09-13T15:25:53.005 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.147+0000 7fc58f7fe640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fc5880386c0 0x7fc58803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:53.005 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.147+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc59006bbc0 con 0x7fc5a0079ee0 2024-09-13T15:25:53.005 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.147+0000 7fc59dd9b640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fc5880386c0 0x7fc58803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:53.005 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.147+0000 7fc59dd9b640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fc5880386c0 0x7fc58803ab80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc5940098c0 tx=0x7fc594006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:53.005 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.151+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fc590043310 con 0x7fc5a0079ee0 2024-09-13T15:25:53.005 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.327+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5a019de70 con 0x7fc5880386c0 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.335+0000 7fc58f7fe640 1 -- 172.21.15.90:0/3200868515 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fc5a019de70 con 0x7fc5880386c0 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fc5880386c0 msgr2=0x7fc58803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fc5880386c0 0x7fc58803ab80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc5940098c0 tx=0x7fc594006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 msgr2=0x7fc5a019a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a019a950 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc590009940 tx=0x7fc590032d40 comp rx=0 tx=0).stop 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 shutdown_connections 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fc5880386c0 0x7fc58803ab80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 --2- 172.21.15.90:0/3200868515 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5a0079ee0 0x7fc5a019a950 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 >> 172.21.15.90:0/3200868515 conn(0x7fc5a0100260 msgr2=0x7fc5a0100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:53.006 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 shutdown_connections 2024-09-13T15:25:53.007 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:52.337+0000 7fc5a4936640 1 -- 172.21.15.90:0/3200868515 wait complete. 2024-09-13T15:25:53.007 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying prometheus service with default placement... 2024-09-13T15:25:53.203 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:52 smithi090 ceph-mon[27811]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:53.203 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:52 smithi090 ceph-mon[27811]: Saving service crash spec with placement * 2024-09-13T15:25:53.203 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:52 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:53.203 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:52 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:54.151 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:53 smithi090 ceph-mon[27811]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "ceph-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:54.151 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:53 smithi090 ceph-mon[27811]: Saving service ceph-exporter spec with placement * 2024-09-13T15:25:54.151 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:53 smithi090 ceph-mon[27811]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:25:54.151 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:53 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:54.151 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:53 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:54.158 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-09-13T15:25:54.158 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5337fc3640 1 Processor -- start 2024-09-13T15:25:54.158 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5337fc3640 1 -- start start 2024-09-13T15:25:54.158 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5337fc3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330107950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:54.158 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5337fc3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330077eb0 con 0x7f5330105560 2024-09-13T15:25:54.158 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5335d5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330107950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5335d5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330107950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56804/0 (socket says 172.21.15.90:56804) 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5335d5f640 1 -- 172.21.15.90:0/563937459 learned_addr learned my addr 172.21.15.90:0/563937459 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.378+0000 7f5335d5f640 1 -- 172.21.15.90:0/563937459 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5330107e90 con 0x7f5330105560 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.379+0000 7f5335d5f640 1 --2- 172.21.15.90:0/563937459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330107950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f532c009820 tx=0x7f532c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8674cddda31e2ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.379+0000 7f5334d5d640 1 -- 172.21.15.90:0/563937459 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f532c02ad40 con 0x7f5330105560 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.379+0000 7f5334d5d640 1 -- 172.21.15.90:0/563937459 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f532c0328d0 con 0x7f5330105560 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.379+0000 7f5334d5d640 1 -- 172.21.15.90:0/563937459 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f532c0306f0 con 0x7f5330105560 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 -- 172.21.15.90:0/563937459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 msgr2=0x7f5330107950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 --2- 172.21.15.90:0/563937459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330107950 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f532c009820 tx=0x7f532c02a270 comp rx=0 tx=0).stop 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 -- 172.21.15.90:0/563937459 shutdown_connections 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 --2- 172.21.15.90:0/563937459 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330107950 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 -- 172.21.15.90:0/563937459 >> 172.21.15.90:0/563937459 conn(0x7f53300ff200 msgr2=0x7f5330101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:54.159 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 -- 172.21.15.90:0/563937459 shutdown_connections 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 -- 172.21.15.90:0/563937459 wait complete. 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.380+0000 7f5337fc3640 1 Processor -- start 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5337fc3640 1 -- start start 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5337fc3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330104de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5337fc3640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5330105320 con 0x7f5330105560 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5335d5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330104de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5335d5f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330104de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56808/0 (socket says 172.21.15.90:56808) 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5335d5f640 1 -- 172.21.15.90:0/4097538434 learned_addr learned my addr 172.21.15.90:0/4097538434 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5335d5f640 1 -- 172.21.15.90:0/4097538434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532c0094d0 con 0x7f5330105560 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5335d5f640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330104de0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f532c0097f0 tx=0x7f532c032710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f532c030c80 con 0x7f5330105560 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f532c030de0 con 0x7f5330105560 2024-09-13T15:25:54.160 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5330103510 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f532c03a560 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.381+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53301039b0 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.382+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f532c03a7e0 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.382+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5330108c90 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.382+0000 7f5322ffd640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f531c038710 0x7f531c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.382+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f532c06ba30 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.383+0000 7f533555e640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f531c038710 0x7f531c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.383+0000 7f533555e640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f531c038710 0x7f531c03abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f53240098c0 tx=0x7f5324006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.387+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f532c039d50 con 0x7f5330105560 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.566+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f5330104160 con 0x7f531c038710 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.570+0000 7f5322ffd640 1 -- 172.21.15.90:0/4097538434 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f5330104160 con 0x7f531c038710 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f531c038710 msgr2=0x7f531c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:54.161 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f531c038710 0x7f531c03abd0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f53240098c0 tx=0x7f5324006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 msgr2=0x7f5330104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330104de0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f532c0097f0 tx=0x7f532c032710 comp rx=0 tx=0).stop 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 shutdown_connections 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f531c038710 0x7f531c03abd0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 --2- 172.21.15.90:0/4097538434 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5330105560 0x7f5330104de0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 >> 172.21.15.90:0/4097538434 conn(0x7f53300ff200 msgr2=0x7f53300ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 shutdown_connections 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:53.573+0000 7f5337fc3640 1 -- 172.21.15.90:0/4097538434 wait complete. 2024-09-13T15:25:54.162 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying grafana service with default placement... 2024-09-13T15:25:55.085 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:54 smithi090 ceph-mon[27811]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:55.085 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:54 smithi090 ceph-mon[27811]: Saving service prometheus spec with placement count:1 2024-09-13T15:25:55.085 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:54 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:55 smithi090 ceph-mon[27811]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:25:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:55 smithi090 ceph-mon[27811]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:55 smithi090 ceph-mon[27811]: Saving service grafana spec with placement count:1 2024-09-13T15:25:56.059 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf2e84c640 1 Processor -- start 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf2e84c640 1 -- start start 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf2e84c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf2e84c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf281054e0 con 0x7faf28104b10 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf27fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf27fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48598/0 (socket says 172.21.15.90:48598) 2024-09-13T15:25:56.060 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf27fff640 1 -- 172.21.15.90:0/2590180234 learned_addr learned my addr 172.21.15.90:0/2590180234 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.564+0000 7faf27fff640 1 -- 172.21.15.90:0/2590180234 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf28105d10 con 0x7faf28104b10 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.565+0000 7faf27fff640 1 --2- 172.21.15.90:0/2590180234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28104f10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf18009820 tx=0x7faf1802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afefa141facc24bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.565+0000 7faf26ffd640 1 -- 172.21.15.90:0/2590180234 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf1802ad40 con 0x7faf28104b10 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.565+0000 7faf26ffd640 1 -- 172.21.15.90:0/2590180234 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faf180328d0 con 0x7faf28104b10 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.565+0000 7faf26ffd640 1 -- 172.21.15.90:0/2590180234 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf180306f0 con 0x7faf28104b10 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 -- 172.21.15.90:0/2590180234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 msgr2=0x7faf28104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 --2- 172.21.15.90:0/2590180234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28104f10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faf18009820 tx=0x7faf1802a270 comp rx=0 tx=0).stop 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 -- 172.21.15.90:0/2590180234 shutdown_connections 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 --2- 172.21.15.90:0/2590180234 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28104f10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 -- 172.21.15.90:0/2590180234 >> 172.21.15.90:0/2590180234 conn(0x7faf281002e0 msgr2=0x7faf28102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:56.061 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 -- 172.21.15.90:0/2590180234 shutdown_connections 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.566+0000 7faf2e84c640 1 -- 172.21.15.90:0/2590180234 wait complete. 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf2e84c640 1 Processor -- start 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf2e84c640 1 -- start start 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf2e84c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf2e84c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf28198c40 con 0x7faf28104b10 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf27fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf27fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48610/0 (socket says 172.21.15.90:48610) 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf27fff640 1 -- 172.21.15.90:0/4172814167 learned_addr learned my addr 172.21.15.90:0/4172814167 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf27fff640 1 -- 172.21.15.90:0/4172814167 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf180094d0 con 0x7faf28104b10 2024-09-13T15:25:56.062 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf27fff640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28198700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf18009940 tx=0x7faf18032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf18030c10 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.567+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faf18030d70 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.568+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf28198e40 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.568+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf1803a560 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.568+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf281992e0 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.568+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7faf1803a820 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.568+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf28108f40 con 0x7faf28104b10 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.569+0000 7faf257fa640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7faf100386c0 0x7faf1003ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:56.063 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.569+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faf1806bba0 con 0x7faf28104b10 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.569+0000 7faf277fe640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7faf100386c0 0x7faf1003ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.569+0000 7faf277fe640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7faf100386c0 0x7faf1003ab80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0098c0 tx=0x7faf1c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.574+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7faf18043310 con 0x7faf28104b10 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.756+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7faf2819bbc0 con 0x7faf100386c0 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.760+0000 7faf257fa640 1 -- 172.21.15.90:0/4172814167 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7faf2819bbc0 con 0x7faf100386c0 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7faf100386c0 msgr2=0x7faf1003ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7faf100386c0 0x7faf1003ab80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0098c0 tx=0x7faf1c006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 msgr2=0x7faf28198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28198700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf18009940 tx=0x7faf18032d40 comp rx=0 tx=0).stop 2024-09-13T15:25:56.064 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 shutdown_connections 2024-09-13T15:25:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7faf100386c0 0x7faf1003ab80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 --2- 172.21.15.90:0/4172814167 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf28104b10 0x7faf28198700 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 >> 172.21.15.90:0/4172814167 conn(0x7faf281002e0 msgr2=0x7faf28100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 shutdown_connections 2024-09-13T15:25:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:54.762+0000 7faf2e84c640 1 -- 172.21.15.90:0/4172814167 wait complete. 2024-09-13T15:25:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying node-exporter service with default placement... 2024-09-13T15:25:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-09-13T15:25:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.497+0000 7f967fbe4640 1 Processor -- start 2024-09-13T15:25:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.498+0000 7f967fbe4640 1 -- start start 2024-09-13T15:25:57.260 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.498+0000 7f967fbe4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678104f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.498+0000 7f967fbe4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96781054e0 con 0x7f9678104b10 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.498+0000 7f967d980640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678104f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.498+0000 7f967d980640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678104f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48612/0 (socket says 172.21.15.90:48612) 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.498+0000 7f967d980640 1 -- 172.21.15.90:0/3150021694 learned_addr learned my addr 172.21.15.90:0/3150021694 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.499+0000 7f967d980640 1 -- 172.21.15.90:0/3150021694 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678105d10 con 0x7f9678104b10 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.499+0000 7f967d980640 1 --2- 172.21.15.90:0/3150021694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678104f10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9668009820 tx=0x7f966802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1619d5459bccd21 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.499+0000 7f967c97e640 1 -- 172.21.15.90:0/3150021694 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966802ad40 con 0x7f9678104b10 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.500+0000 7f967c97e640 1 -- 172.21.15.90:0/3150021694 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f96680328d0 con 0x7f9678104b10 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.500+0000 7f967c97e640 1 -- 172.21.15.90:0/3150021694 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f96680306f0 con 0x7f9678104b10 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.500+0000 7f967fbe4640 1 -- 172.21.15.90:0/3150021694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 msgr2=0x7f9678104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:57.261 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.500+0000 7f967fbe4640 1 --2- 172.21.15.90:0/3150021694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678104f10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9668009820 tx=0x7f966802a270 comp rx=0 tx=0).stop 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 -- 172.21.15.90:0/3150021694 shutdown_connections 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 --2- 172.21.15.90:0/3150021694 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678104f10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 -- 172.21.15.90:0/3150021694 >> 172.21.15.90:0/3150021694 conn(0x7f96781002e0 msgr2=0x7f9678102700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 -- 172.21.15.90:0/3150021694 shutdown_connections 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 -- 172.21.15.90:0/3150021694 wait complete. 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 Processor -- start 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 -- start start 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.501+0000 7f967fbe4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9678198c40 con 0x7f9678104b10 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967d980640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:57.262 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967d980640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678198700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48616/0 (socket says 172.21.15.90:48616) 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967d980640 1 -- 172.21.15.90:0/801245671 learned_addr learned my addr 172.21.15.90:0/801245671 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967d980640 1 -- 172.21.15.90:0/801245671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96680094d0 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967d980640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678198700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f96680097f0 tx=0x7f96680326f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9668030c80 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9668030de0 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f966803a540 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9678198e40 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.502+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96781992e0 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.503+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f966803a6a0 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.503+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9678109090 con 0x7f9678104b10 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.503+0000 7f966effd640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f9664038710 0x7f966403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:57.263 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.504+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f966806ba40 con 0x7f9678104b10 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.504+0000 7f967d17f640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f9664038710 0x7f966403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.504+0000 7f967d17f640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f9664038710 0x7f966403abd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f96740098c0 tx=0x7f9674006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.508+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f966806de60 con 0x7f9678104b10 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.690+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f9678101480 con 0x7f9664038710 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.695+0000 7f966effd640 1 -- 172.21.15.90:0/801245671 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f9678101480 con 0x7f9664038710 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.697+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f9664038710 msgr2=0x7f966403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:57.264 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.697+0000 7f967fbe4640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f9664038710 0x7f966403abd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f96740098c0 tx=0x7f9674006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 msgr2=0x7f9678198700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678198700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f96680097f0 tx=0x7f96680326f0 comp rx=0 tx=0).stop 2024-09-13T15:25:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 shutdown_connections 2024-09-13T15:25:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f9664038710 0x7f966403abd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 --2- 172.21.15.90:0/801245671 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9678104b10 0x7f9678198700 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:57.265 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 >> 172.21.15.90:0/801245671 conn(0x7f96781002e0 msgr2=0x7f9678100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:57.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 shutdown_connections 2024-09-13T15:25:57.266 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:56.698+0000 7f967fbe4640 1 -- 172.21.15.90:0/801245671 wait complete. 2024-09-13T15:25:57.266 INFO:teuthology.orchestra.run.smithi090.stdout:Deploying alertmanager service with default placement... 2024-09-13T15:25:57.857 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:57 smithi090 ceph-mon[27811]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:57.857 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:57 smithi090 ceph-mon[27811]: Saving service node-exporter spec with placement * 2024-09-13T15:25:57.857 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:57 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:57.858 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:57 smithi090 ceph-mon[27811]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:25:58.370 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-09-13T15:25:58.370 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.673+0000 7f23ffcaa640 1 Processor -- start 2024-09-13T15:25:58.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.673+0000 7f23ffcaa640 1 -- start start 2024-09-13T15:25:58.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.674+0000 7f23ffcaa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8107980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:58.371 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.674+0000 7f23ffcaa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8077e50 con 0x7f23f8105590 2024-09-13T15:25:58.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.674+0000 7f23fda46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8107980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:58.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.674+0000 7f23fda46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8107980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48624/0 (socket says 172.21.15.90:48624) 2024-09-13T15:25:58.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.674+0000 7f23fda46640 1 -- 172.21.15.90:0/1830934387 learned_addr learned my addr 172.21.15.90:0/1830934387 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:58.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.674+0000 7f23fda46640 1 -- 172.21.15.90:0/1830934387 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8107ec0 con 0x7f23f8105590 2024-09-13T15:25:58.372 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23fda46640 1 --2- 172.21.15.90:0/1830934387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8107980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009e80 tx=0x7f23e802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c603e48dbb17d501 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23fca44640 1 -- 172.21.15.90:0/1830934387 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e8031400 con 0x7f23f8105590 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23fca44640 1 -- 172.21.15.90:0/1830934387 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f23e802d050 con 0x7f23f8105590 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23fca44640 1 -- 172.21.15.90:0/1830934387 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e8007720 con 0x7f23f8105590 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23ffcaa640 1 -- 172.21.15.90:0/1830934387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 msgr2=0x7f23f8107980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23ffcaa640 1 --2- 172.21.15.90:0/1830934387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8107980 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009e80 tx=0x7f23e802a7a0 comp rx=0 tx=0).stop 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23ffcaa640 1 -- 172.21.15.90:0/1830934387 shutdown_connections 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23ffcaa640 1 --2- 172.21.15.90:0/1830934387 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8107980 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.675+0000 7f23ffcaa640 1 -- 172.21.15.90:0/1830934387 >> 172.21.15.90:0/1830934387 conn(0x7f23f80ff110 msgr2=0x7f23f8101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:58.373 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23ffcaa640 1 -- 172.21.15.90:0/1830934387 shutdown_connections 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23ffcaa640 1 -- 172.21.15.90:0/1830934387 wait complete. 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23ffcaa640 1 Processor -- start 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23ffcaa640 1 -- start start 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23ffcaa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23ffcaa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8198c70 con 0x7f23f8105590 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23fda46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:58.374 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.676+0000 7f23fda46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48626/0 (socket says 172.21.15.90:48626) 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23fda46640 1 -- 172.21.15.90:0/301259091 learned_addr learned my addr 172.21.15.90:0/301259091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23fda46640 1 -- 172.21.15.90:0/301259091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e8009b30 con 0x7f23f8105590 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23fda46640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8198730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23e8031660 tx=0x7f23e8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e8031c00 con 0x7f23f8105590 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f23e8031d60 con 0x7f23f8105590 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23e80396b0 con 0x7f23f8105590 2024-09-13T15:25:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f8198e70 con 0x7f23f8105590 2024-09-13T15:25:58.376 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.677+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f8199310 con 0x7f23f8105590 2024-09-13T15:25:58.376 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.678+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f23e8038040 con 0x7f23f8105590 2024-09-13T15:25:58.376 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.678+0000 7f23f6ffd640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f23e0038710 0x7f23e003abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:58.376 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.678+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f8108c60 con 0x7f23f8105590 2024-09-13T15:25:58.376 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.678+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f23e806b7d0 con 0x7f23f8105590 2024-09-13T15:25:58.376 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.678+0000 7f23fd245640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f23e0038710 0x7f23e003abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.679+0000 7f23fd245640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f23e0038710 0x7f23e003abd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f23ec0098c0 tx=0x7f23ec006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.683+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f23e8041070 con 0x7f23f8105590 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.854+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f23f8100450 con 0x7f23e0038710 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.859+0000 7f23f6ffd640 1 -- 172.21.15.90:0/301259091 <== mgr.14118 v2:172.21.15.90:6800/3565908774 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f23f8100450 con 0x7f23e0038710 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.861+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f23e0038710 msgr2=0x7f23e003abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.861+0000 7f23ffcaa640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f23e0038710 0x7f23e003abd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f23ec0098c0 tx=0x7f23ec006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:58.377 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.861+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 msgr2=0x7f23f8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.861+0000 7f23ffcaa640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8198730 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f23e8031660 tx=0x7f23e8007f90 comp rx=0 tx=0).stop 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.862+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 shutdown_connections 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.862+0000 7f23ffcaa640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f23e0038710 0x7f23e003abd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.862+0000 7f23ffcaa640 1 --2- 172.21.15.90:0/301259091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23f8105590 0x7f23f8198730 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.862+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 >> 172.21.15.90:0/301259091 conn(0x7f23f80ff110 msgr2=0x7f23f80ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.862+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 shutdown_connections 2024-09-13T15:25:58.378 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:57.862+0000 7f23ffcaa640 1 -- 172.21.15.90:0/301259091 wait complete. 2024-09-13T15:25:59.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:58 smithi090 ceph-mon[27811]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:25:59.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:58 smithi090 ceph-mon[27811]: Saving service alertmanager spec with placement count:1 2024-09-13T15:25:59.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:58 smithi090 ceph-mon[27811]: from='mgr.14118 172.21.15.90:0/472571064' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:25:59.517 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.736+0000 7fafc0ae8640 1 Processor -- start 2024-09-13T15:25:59.517 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.736+0000 7fafc0ae8640 1 -- start start 2024-09-13T15:25:59.517 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.737+0000 7fafc0ae8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc107960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:59.517 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.737+0000 7fafc0ae8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc077e70 con 0x7fafbc105570 2024-09-13T15:25:59.517 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.737+0000 7fafba59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc107960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.737+0000 7fafba59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc107960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48636/0 (socket says 172.21.15.90:48636) 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.737+0000 7fafba59c640 1 -- 172.21.15.90:0/3767085321 learned_addr learned my addr 172.21.15.90:0/3767085321 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.737+0000 7fafba59c640 1 -- 172.21.15.90:0/3767085321 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc107ea0 con 0x7fafbc105570 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.738+0000 7fafba59c640 1 --2- 172.21.15.90:0/3767085321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc107960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fafac009820 tx=0x7fafac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5b3aac9767e7e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.738+0000 7fafb9d9b640 1 -- 172.21.15.90:0/3767085321 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafac02ad40 con 0x7fafbc105570 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.738+0000 7fafb9d9b640 1 -- 172.21.15.90:0/3767085321 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fafac0328d0 con 0x7fafbc105570 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.738+0000 7fafb9d9b640 1 -- 172.21.15.90:0/3767085321 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafac0306f0 con 0x7fafbc105570 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.738+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3767085321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 msgr2=0x7fafbc107960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.738+0000 7fafc0ae8640 1 --2- 172.21.15.90:0/3767085321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc107960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fafac009820 tx=0x7fafac02a270 comp rx=0 tx=0).stop 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.739+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3767085321 shutdown_connections 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.739+0000 7fafc0ae8640 1 --2- 172.21.15.90:0/3767085321 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc107960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.739+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3767085321 >> 172.21.15.90:0/3767085321 conn(0x7fafbc0ff110 msgr2=0x7fafbc101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:59.518 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.739+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3767085321 shutdown_connections 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.739+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3767085321 wait complete. 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.739+0000 7fafc0ae8640 1 Processor -- start 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafc0ae8640 1 -- start start 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafc0ae8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafc0ae8640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc198c80 con 0x7fafbc105570 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafba59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafba59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48650/0 (socket says 172.21.15.90:48650) 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafba59c640 1 -- 172.21.15.90:0/3051891917 learned_addr learned my addr 172.21.15.90:0/3051891917 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafba59c640 1 -- 172.21.15.90:0/3051891917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0094d0 con 0x7fafbc105570 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafba59c640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc198740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafac032d40 tx=0x7fafac032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafac030c10 con 0x7fafbc105570 2024-09-13T15:25:59.519 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.740+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc198e80 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.741+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fafac030d70 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.741+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafac03a520 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.741+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc199320 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.741+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fafac03a7e0 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.742+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafbc108c40 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.742+0000 7fafb3fff640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fafa80386c0 0x7fafa803ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.742+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fafac06ba20 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.742+0000 7fafb1d9b640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fafa80386c0 0x7fafa803ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.742+0000 7fafb1d9b640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fafa80386c0 0x7fafa803ab80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fafa40098c0 tx=0x7fafa4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.746+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fafac039d50 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.910+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fafbc19bc60 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.913+0000 7fafb3fff640 1 -- 172.21.15.90:0/3051891917 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fafac039310 con 0x7fafbc105570 2024-09-13T15:25:59.520 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fafa80386c0 msgr2=0x7fafa803ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fafa80386c0 0x7fafa803ab80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fafa40098c0 tx=0x7fafa4006db0 comp rx=0 tx=0).stop 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 msgr2=0x7fafbc198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc198740 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafac032d40 tx=0x7fafac032d70 comp rx=0 tx=0).stop 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 shutdown_connections 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fafa80386c0 0x7fafa803ab80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 --2- 172.21.15.90:0/3051891917 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafbc105570 0x7fafbc198740 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 >> 172.21.15.90:0/3051891917 conn(0x7fafbc0ff110 msgr2=0x7fafbc0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 shutdown_connections 2024-09-13T15:25:59.521 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:25:58.916+0000 7fafc0ae8640 1 -- 172.21.15.90:0/3051891917 wait complete. 2024-09-13T15:26:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:59 smithi090 ceph-mon[27811]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:25:59 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3051891917' entity='client.admin' 2024-09-13T15:26:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7feddc87b640 1 Processor -- start 2024-09-13T15:26:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7feddc87b640 1 -- start start 2024-09-13T15:26:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7feddc87b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7feddc87b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd81054d0 con 0x7fedd8104b00 2024-09-13T15:26:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7fedd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:00.868 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7fedd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48662/0 (socket says 172.21.15.90:48662) 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.118+0000 7fedd659c640 1 -- 172.21.15.90:0/836438547 learned_addr learned my addr 172.21.15.90:0/836438547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.119+0000 7fedd659c640 1 -- 172.21.15.90:0/836438547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd8105d00 con 0x7fedd8104b00 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.119+0000 7fedd659c640 1 --2- 172.21.15.90:0/836438547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8104f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009820 tx=0x7fedc802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25005071db44d0c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.119+0000 7fedd5d9b640 1 -- 172.21.15.90:0/836438547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc802ad40 con 0x7fedd8104b00 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7fedd5d9b640 1 -- 172.21.15.90:0/836438547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fedc80328d0 con 0x7fedd8104b00 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7fedd5d9b640 1 -- 172.21.15.90:0/836438547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc80306f0 con 0x7fedd8104b00 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 -- 172.21.15.90:0/836438547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 msgr2=0x7fedd8104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 --2- 172.21.15.90:0/836438547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8104f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009820 tx=0x7fedc802a270 comp rx=0 tx=0).stop 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 -- 172.21.15.90:0/836438547 shutdown_connections 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 --2- 172.21.15.90:0/836438547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8104f00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:00.869 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 -- 172.21.15.90:0/836438547 >> 172.21.15.90:0/836438547 conn(0x7fedd8100270 msgr2=0x7fedd81026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 -- 172.21.15.90:0/836438547 shutdown_connections 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.120+0000 7feddc87b640 1 -- 172.21.15.90:0/836438547 wait complete. 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7feddc87b640 1 Processor -- start 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7feddc87b640 1 -- start start 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7feddc87b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7feddc87b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd8198c70 con 0x7fedd8104b00 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7fedd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7fedd659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48666/0 (socket says 172.21.15.90:48666) 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.121+0000 7fedd659c640 1 -- 172.21.15.90:0/2199568357 learned_addr learned my addr 172.21.15.90:0/2199568357 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7fedd659c640 1 -- 172.21.15.90:0/2199568357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc80094d0 con 0x7fedd8104b00 2024-09-13T15:26:00.870 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7fedd659c640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8198730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009940 tx=0x7fedc8032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc8030c10 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fedc8030d70 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd8198e70 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fedc803a560 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.122+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd8199310 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.123+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7fedc803a7d0 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.123+0000 7fedcf7fe640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedc4038710 0x7fedc403abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.123+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fedc806ba20 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.123+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd8108f30 con 0x7fedd8104b00 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.123+0000 7fedcffff640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedc4038710 0x7fedc403abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:00.871 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.124+0000 7fedcffff640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedc4038710 0x7fedc403abd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fedc00098c0 tx=0x7fedc0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.128+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fedc8046460 con 0x7fedd8104b00 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.290+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fedd8109170 con 0x7fedd8104b00 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.293+0000 7fedcf7fe640 1 -- 172.21.15.90:0/2199568357 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fedc8039310 con 0x7fedd8104b00 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.295+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedc4038710 msgr2=0x7fedc403abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.295+0000 7feddc87b640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedc4038710 0x7fedc403abd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fedc00098c0 tx=0x7fedc0006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.295+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 msgr2=0x7fedd8198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.295+0000 7feddc87b640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8198730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009940 tx=0x7fedc8032d40 comp rx=0 tx=0).stop 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.296+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 shutdown_connections 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.296+0000 7feddc87b640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fedc4038710 0x7fedc403abd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.296+0000 7feddc87b640 1 --2- 172.21.15.90:0/2199568357 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fedd8104b00 0x7fedd8198730 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:00.872 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.296+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 >> 172.21.15.90:0/2199568357 conn(0x7fedd8100270 msgr2=0x7fedd8100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.296+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 shutdown_connections 2024-09-13T15:26:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:00.296+0000 7feddc87b640 1 -- 172.21.15.90:0/2199568357 wait complete. 2024-09-13T15:26:00.873 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling the dashboard module... 2024-09-13T15:26:01.468 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:01 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2199568357' entity='client.admin' 2024-09-13T15:26:01.468 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:01 smithi090 ceph-mon[27811]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:02.653 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:02 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/65044588' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-13T15:26:02.951 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.250+0000 7f7f3c547640 1 Processor -- start 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.250+0000 7f7f3c547640 1 -- start start 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.250+0000 7f7f3c547640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341078e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.250+0000 7f7f3c547640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f34077eb0 con 0x7f7f341054f0 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.251+0000 7f7f3a2e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341078e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.251+0000 7f7f3a2e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341078e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48680/0 (socket says 172.21.15.90:48680) 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.251+0000 7f7f3a2e3640 1 -- 172.21.15.90:0/2164764029 learned_addr learned my addr 172.21.15.90:0/2164764029 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.251+0000 7f7f3a2e3640 1 -- 172.21.15.90:0/2164764029 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f34107e20 con 0x7f7f341054f0 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.251+0000 7f7f3a2e3640 1 --2- 172.21.15.90:0/2164764029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341078e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009e80 tx=0x7f7f2402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28b0f51050c48de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f392e1640 1 -- 172.21.15.90:0/2164764029 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f24031400 con 0x7f7f341054f0 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f392e1640 1 -- 172.21.15.90:0/2164764029 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7f2402d050 con 0x7f7f341054f0 2024-09-13T15:26:02.952 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f392e1640 1 -- 172.21.15.90:0/2164764029 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f24007720 con 0x7f7f341054f0 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 -- 172.21.15.90:0/2164764029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 msgr2=0x7f7f341078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 --2- 172.21.15.90:0/2164764029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341078e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009e80 tx=0x7f7f2402a7a0 comp rx=0 tx=0).stop 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 -- 172.21.15.90:0/2164764029 shutdown_connections 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 --2- 172.21.15.90:0/2164764029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341078e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 -- 172.21.15.90:0/2164764029 >> 172.21.15.90:0/2164764029 conn(0x7f7f340ff130 msgr2=0x7f7f34101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 -- 172.21.15.90:0/2164764029 shutdown_connections 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.252+0000 7f7f3c547640 1 -- 172.21.15.90:0/2164764029 wait complete. 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3c547640 1 Processor -- start 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3c547640 1 -- start start 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3c547640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3c547640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f34198be0 con 0x7f7f341054f0 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3a2e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3a2e3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48694/0 (socket says 172.21.15.90:48694) 2024-09-13T15:26:02.953 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.253+0000 7f7f3a2e3640 1 -- 172.21.15.90:0/65044588 learned_addr learned my addr 172.21.15.90:0/65044588 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f3a2e3640 1 -- 172.21.15.90:0/65044588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24009b30 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f3a2e3640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341986a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009e50 tx=0x7f7f24031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f24031bd0 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7f24031d30 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f24044550 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f34198de0 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.254+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f34199280 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.255+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 8) v1 ==== 45556+0+0 (secure 0 0 0) 0x7f7f24038040 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.255+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f34108c20 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.255+0000 7f7f337fe640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7f1c038760 0x7f7f1c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.255+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7f2406bca0 con 0x7f7f341054f0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.255+0000 7f7f39ae2640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7f1c038760 0x7f7f1c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:02.954 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.256+0000 7f7f39ae2640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7f1c038760 0x7f7f1c03ac20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f280098c0 tx=0x7f7f28006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.260+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f7f2404db30 con 0x7f7f341054f0 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:01.466+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f7f34108e60 con 0x7f7f341054f0 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.298+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f7f2403a400 con 0x7f7f341054f0 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.300+0000 7f7f337fe640 1 -- 172.21.15.90:0/65044588 <== mon.0 v2:172.21.15.90:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7f24046b30 con 0x7f7f341054f0 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7f1c038760 msgr2=0x7f7f1c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7f1c038760 0x7f7f1c03ac20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7f280098c0 tx=0x7f7f28006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 msgr2=0x7f7f341986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341986a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7f24009e50 tx=0x7f7f24031630 comp rx=0 tx=0).stop 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 shutdown_connections 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7f7f1c038760 0x7f7f1c03ac20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 --2- 172.21.15.90:0/65044588 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7f341054f0 0x7f7f341986a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 >> 172.21.15.90:0/65044588 conn(0x7f7f340ff130 msgr2=0x7f7f340ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:02.955 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 shutdown_connections 2024-09-13T15:26:02.956 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:02.303+0000 7f7f3c547640 1 -- 172.21.15.90:0/65044588 wait complete. 2024-09-13T15:26:03.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:03 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/65044588' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-13T15:26:03.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:03 smithi090 ceph-mon[27811]: mgrmap e9: smithi090.fyvvpd(active, since 35s) 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "active_name": "smithi090.fyvvpd", 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.349+0000 7ff294768640 1 Processor -- start 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.349+0000 7ff294768640 1 -- start start 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.350+0000 7ff294768640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c107960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.350+0000 7ff294768640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c077e70 con 0x7ff28c105570 2024-09-13T15:26:04.137 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.350+0000 7ff292504640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c107960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.350+0000 7ff292504640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c107960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48728/0 (socket says 172.21.15.90:48728) 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.350+0000 7ff292504640 1 -- 172.21.15.90:0/903160375 learned_addr learned my addr 172.21.15.90:0/903160375 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.350+0000 7ff292504640 1 -- 172.21.15.90:0/903160375 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c107ea0 con 0x7ff28c105570 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.351+0000 7ff292504640 1 --2- 172.21.15.90:0/903160375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c107960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff288009820 tx=0x7ff28802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd449514ce56873 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.351+0000 7ff291502640 1 -- 172.21.15.90:0/903160375 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff28802ad40 con 0x7ff28c105570 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.351+0000 7ff291502640 1 -- 172.21.15.90:0/903160375 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff2880328d0 con 0x7ff28c105570 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.351+0000 7ff291502640 1 -- 172.21.15.90:0/903160375 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2880306f0 con 0x7ff28c105570 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 -- 172.21.15.90:0/903160375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 msgr2=0x7ff28c107960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 --2- 172.21.15.90:0/903160375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c107960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff288009820 tx=0x7ff28802a270 comp rx=0 tx=0).stop 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 -- 172.21.15.90:0/903160375 shutdown_connections 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 --2- 172.21.15.90:0/903160375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c107960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 -- 172.21.15.90:0/903160375 >> 172.21.15.90:0/903160375 conn(0x7ff28c0ff110 msgr2=0x7ff28c101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:04.138 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 -- 172.21.15.90:0/903160375 shutdown_connections 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 -- 172.21.15.90:0/903160375 wait complete. 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.352+0000 7ff294768640 1 Processor -- start 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff294768640 1 -- start start 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff294768640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff294768640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c198c80 con 0x7ff28c105570 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff292504640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff292504640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48744/0 (socket says 172.21.15.90:48744) 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff292504640 1 -- 172.21.15.90:0/1887307568 learned_addr learned my addr 172.21.15.90:0/1887307568 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff292504640 1 -- 172.21.15.90:0/1887307568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2880094d0 con 0x7ff28c105570 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff292504640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c198740 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2880097f0 tx=0x7ff288032710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.353+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff288030c80 con 0x7ff28c105570 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.354+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff288030de0 con 0x7ff28c105570 2024-09-13T15:26:04.139 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.354+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c198e80 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.354+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c199320 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.354+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff28803a560 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.355+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c108d90 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.355+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7ff28803ca20 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.355+0000 7ff27f7fe640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 0x7ff27803ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.355+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff28806b980 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.355+0000 7ff291d03640 1 -- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 msgr2=0x7ff27803ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3565908774 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.355+0000 7ff291d03640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 0x7ff27803ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.359+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7ff288031d70 con 0x7ff28c105570 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.556+0000 7ff291d03640 1 -- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 msgr2=0x7ff27803ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3565908774 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.556+0000 7ff291d03640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 0x7ff27803ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T15:26:04.140 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.568+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff28c19bc00 con 0x7ff28c105570 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.568+0000 7ff27f7fe640 1 -- 172.21.15.90:0/1887307568 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7ff288031b90 con 0x7ff28c105570 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 msgr2=0x7ff27803ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 0x7ff27803ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 msgr2=0x7ff28c198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c198740 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff2880097f0 tx=0x7ff288032710 comp rx=0 tx=0).stop 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 shutdown_connections 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7ff278038760 0x7ff27803ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 --2- 172.21.15.90:0/1887307568 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff28c105570 0x7ff28c198740 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 >> 172.21.15.90:0/1887307568 conn(0x7ff28c0ff110 msgr2=0x7ff28c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 shutdown_connections 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:03.570+0000 7ff294768640 1 -- 172.21.15.90:0/1887307568 wait complete. 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for the mgr to restart... 2024-09-13T15:26:04.141 INFO:teuthology.orchestra.run.smithi090.stdout:Waiting for mgr epoch 9... 2024-09-13T15:26:04.432 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:04 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1887307568' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: Active manager daemon smithi090.fyvvpd restarted 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: Activating manager daemon smithi090.fyvvpd 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: osdmap e3: 0 total, 0 up, 0 in 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: mgrmap e10: smithi090.fyvvpd(active, starting, since 0.00589942s) 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr metadata", "who": "smithi090.fyvvpd", "id": "smithi090.fyvvpd"}]: dispatch 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: Manager daemon smithi090.fyvvpd is now available 2024-09-13T15:26:07.414 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:26:07.415 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/mirror_snapshot_schedule"}]: dispatch 2024-09-13T15:26:07.415 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:07 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/trash_purge_schedule"}]: dispatch 2024-09-13T15:26:08.585 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout { 2024-09-13T15:26:08.585 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-13T15:26:08.585 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout } 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.508+0000 7fb17a6ce640 1 Processor -- start 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.508+0000 7fb17a6ce640 1 -- start start 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.509+0000 7fb17a6ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174104f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.509+0000 7fb17a6ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1741054d0 con 0x7fb174104b00 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.509+0000 7fb173fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.509+0000 7fb173fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174104f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47156/0 (socket says 172.21.15.90:47156) 2024-09-13T15:26:08.586 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.509+0000 7fb173fff640 1 -- 172.21.15.90:0/3145570281 learned_addr learned my addr 172.21.15.90:0/3145570281 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.509+0000 7fb173fff640 1 -- 172.21.15.90:0/3145570281 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb174105d00 con 0x7fb174104b00 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.510+0000 7fb173fff640 1 --2- 172.21.15.90:0/3145570281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174104f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb164009820 tx=0x7fb16402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87cf643de2410ac6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.510+0000 7fb1737fe640 1 -- 172.21.15.90:0/3145570281 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb16402ad40 con 0x7fb174104b00 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.510+0000 7fb1737fe640 1 -- 172.21.15.90:0/3145570281 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb1640328d0 con 0x7fb174104b00 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.510+0000 7fb1737fe640 1 -- 172.21.15.90:0/3145570281 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1640306f0 con 0x7fb174104b00 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 -- 172.21.15.90:0/3145570281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 msgr2=0x7fb174104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/3145570281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174104f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb164009820 tx=0x7fb16402a270 comp rx=0 tx=0).stop 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 -- 172.21.15.90:0/3145570281 shutdown_connections 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/3145570281 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174104f00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 -- 172.21.15.90:0/3145570281 >> 172.21.15.90:0/3145570281 conn(0x7fb1741002d0 msgr2=0x7fb1741026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 -- 172.21.15.90:0/3145570281 shutdown_connections 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 -- 172.21.15.90:0/3145570281 wait complete. 2024-09-13T15:26:08.587 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 Processor -- start 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.511+0000 7fb17a6ce640 1 -- start start 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb17a6ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb17a6ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17407a0b0 con 0x7fb174104b00 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb173fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb173fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174079b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47172/0 (socket says 172.21.15.90:47172) 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb173fff640 1 -- 172.21.15.90:0/2423423001 learned_addr learned my addr 172.21.15.90:0/2423423001 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb173fff640 1 -- 172.21.15.90:0/2423423001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1640094d0 con 0x7fb174104b00 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb173fff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174079b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb164009940 tx=0x7fb164032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb164030c10 con 0x7fb174104b00 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb164030d70 con 0x7fb174104b00 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1740782a0 con 0x7fb174104b00 2024-09-13T15:26:08.588 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb16403a560 con 0x7fb174104b00 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.512+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb174078740 con 0x7fb174104b00 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.513+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 9) v1 ==== 45569+0+0 (secure 0 0 0) 0x7fb16403a7d0 con 0x7fb174104b00 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.513+0000 7fb171ffb640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.513+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 --> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb16003b330 con 0x7fb160038760 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.513+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb16406bab0 con 0x7fb174104b00 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.513+0000 7fb16bfff640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 msgr2=0x7fb16003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3565908774 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.513+0000 7fb16bfff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.714+0000 7fb16bfff640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 msgr2=0x7fb16003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3565908774 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:04.714+0000 7fb16bfff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:05.115+0000 7fb16bfff640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 msgr2=0x7fb16003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3565908774 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:05.115+0000 7fb16bfff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-13T15:26:08.589 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:05.916+0000 7fb16bfff640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 msgr2=0x7fb16003ac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3565908774 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:05.916+0000 7fb16bfff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:07.055+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mgrmap(e 10) v1 ==== 45336+0+0 (secure 0 0 0) 0x7fb16406a8e0 con 0x7fb174104b00 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:07.055+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 msgr2=0x7fb16003ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:07.055+0000 7fb171ffb640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.052+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 11) v1 ==== 45463+0+0 (secure 0 0 0) 0x7fb164046980 con 0x7fb174104b00 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.052+0000 7fb171ffb640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb16003bfb0 0x7fb16003e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.052+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb16003b330 con 0x7fb16003bfb0 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.053+0000 7fb16bfff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb16003bfb0 0x7fb16003e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.054+0000 7fb16bfff640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb16003bfb0 0x7fb16003e3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb15c003980 tx=0x7fb15c0092b0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.054+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fb16003b330 con 0x7fb16003bfb0 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb174104f80 con 0x7fb16003bfb0 2024-09-13T15:26:08.590 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb171ffb640 1 -- 172.21.15.90:0/2423423001 <== mgr.14162 v2:172.21.15.90:6800/3319538530 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb174104f80 con 0x7fb16003bfb0 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb16003bfb0 msgr2=0x7fb16003e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb16003bfb0 0x7fb16003e3a0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb15c003980 tx=0x7fb15c0092b0 comp rx=0 tx=0).stop 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 msgr2=0x7fb174079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174079b70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb164009940 tx=0x7fb164032d40 comp rx=0 tx=0).stop 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 shutdown_connections 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb16003bfb0 0x7fb16003e3a0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:6800/3565908774,v1:172.21.15.90:6801/3565908774] conn(0x7fb160038760 0x7fb16003ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 --2- 172.21.15.90:0/2423423001 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb174104b00 0x7fb174079b70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.060+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 >> 172.21.15.90:0/2423423001 conn(0x7fb1741002d0 msgr2=0x7fb174100d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.061+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 shutdown_connections 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:08.061+0000 7fb17a6ce640 1 -- 172.21.15.90:0/2423423001 wait complete. 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:mgr epoch 9 is available 2024-09-13T15:26:08.591 INFO:teuthology.orchestra.run.smithi090.stdout:Generating a dashboard self-signed certificate... 2024-09-13T15:26:09.322 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:09 smithi090 ceph-mon[27811]: mgrmap e11: smithi090.fyvvpd(active, since 1.00577s) 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:09 smithi090 ceph-mon[27811]: [13/Sep/2024:15:26:08] ENGINE Bus STARTING 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:09 smithi090 ceph-mon[27811]: [13/Sep/2024:15:26:08] ENGINE Serving on https://172.21.15.90:7150 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:09 smithi090 ceph-mon[27811]: [13/Sep/2024:15:26:08] ENGINE Error in HTTPServer.serve 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Traceback (most recent call last): 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._connections.run(self.expiration_interval) 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._run(expiration_interval) 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T15:26:09.323 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: s = self.context.wrap_socket( 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: return self.sslsocket_class._create( 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self.do_handshake() 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._sslobj.do_handshake() 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T15:26:09.324 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:09 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:09.874 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.016+0000 7fd59eab9640 1 Processor -- start 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59eab9640 1 -- start start 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59eab9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598107980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59eab9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598077e50 con 0x7fd598105590 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59c855640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598107980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59c855640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598107980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47256/0 (socket says 172.21.15.90:47256) 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59c855640 1 -- 172.21.15.90:0/1548854600 learned_addr learned my addr 172.21.15.90:0/1548854600 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.017+0000 7fd59c855640 1 -- 172.21.15.90:0/1548854600 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598107ec0 con 0x7fd598105590 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.018+0000 7fd59c855640 1 --2- 172.21.15.90:0/1548854600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598107980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009820 tx=0x7fd58c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcdc4c0d52c0e26 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.018+0000 7fd5977fe640 1 -- 172.21.15.90:0/1548854600 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c02ad40 con 0x7fd598105590 2024-09-13T15:26:09.875 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.018+0000 7fd5977fe640 1 -- 172.21.15.90:0/1548854600 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd58c0328d0 con 0x7fd598105590 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.018+0000 7fd5977fe640 1 -- 172.21.15.90:0/1548854600 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c0306f0 con 0x7fd598105590 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 -- 172.21.15.90:0/1548854600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 msgr2=0x7fd598107980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 --2- 172.21.15.90:0/1548854600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598107980 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd58c009820 tx=0x7fd58c02a270 comp rx=0 tx=0).stop 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 -- 172.21.15.90:0/1548854600 shutdown_connections 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 --2- 172.21.15.90:0/1548854600 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598107980 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 -- 172.21.15.90:0/1548854600 >> 172.21.15.90:0/1548854600 conn(0x7fd5980ff110 msgr2=0x7fd598101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 -- 172.21.15.90:0/1548854600 shutdown_connections 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.019+0000 7fd59eab9640 1 -- 172.21.15.90:0/1548854600 wait complete. 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59eab9640 1 Processor -- start 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59eab9640 1 -- start start 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59eab9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598198730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59eab9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598198c70 con 0x7fd598105590 2024-09-13T15:26:09.876 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59c855640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598198730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59c855640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598198730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47258/0 (socket says 172.21.15.90:47258) 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59c855640 1 -- 172.21.15.90:0/1620492820 learned_addr learned my addr 172.21.15.90:0/1620492820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59c855640 1 -- 172.21.15.90:0/1620492820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c0094d0 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59c855640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598198730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd58c032d40 tx=0x7fd58c032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c030c10 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd58c030d70 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd598198e70 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.020+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c03a520 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.021+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd598199310 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.021+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 11) v1 ==== 45463+0+0 (secure 0 0 0) 0x7fd58c03a7e0 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.022+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd598108c60 con 0x7fd598105590 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.022+0000 7fd595ffb640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd5880385f0 0x7fd58803aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:09.877 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.022+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd58c06c820 con 0x7fd598105590 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.022+0000 7fd597fff640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd5880385f0 0x7fd58803aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.022+0000 7fd597fff640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd5880385f0 0x7fd58803aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5840098c0 tx=0x7fd584006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.030+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd58c043310 con 0x7fd598105590 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.217+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd598100450 con 0x7fd5880385f0 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.320+0000 7fd595ffb640 1 -- 172.21.15.90:0/1620492820 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd598100450 con 0x7fd5880385f0 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd5880385f0 msgr2=0x7fd58803aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd5880385f0 0x7fd58803aab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd5840098c0 tx=0x7fd584006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 msgr2=0x7fd598198730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598198730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd58c032d40 tx=0x7fd58c032d70 comp rx=0 tx=0).stop 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 shutdown_connections 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd5880385f0 0x7fd58803aab0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 --2- 172.21.15.90:0/1620492820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd598105590 0x7fd598198730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 >> 172.21.15.90:0/1620492820 conn(0x7fd5980ff110 msgr2=0x7fd5980ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:09.878 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 shutdown_connections 2024-09-13T15:26:09.879 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:09.322+0000 7fd59eab9640 1 -- 172.21.15.90:0/1620492820 wait complete. 2024-09-13T15:26:09.879 INFO:teuthology.orchestra.run.smithi090.stdout:Creating initial admin user... 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: [13/Sep/2024:15:26:08] ENGINE Serving on http://172.21.15.90:8765 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: [13/Sep/2024:15:26:08] ENGINE Bus STARTED 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:10.416 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:10 smithi090 ceph-mon[27811]: mgrmap e12: smithi090.fyvvpd(active, since 2s) 2024-09-13T15:26:12.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$7uo8A78BL5oGxqtJlH2MOeqNeEGLB.0gZE68Z/WVqnxwGTxWIfhgG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726241171, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-13T15:26:12.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.206+0000 7f9239ee2640 1 Processor -- start 2024-09-13T15:26:12.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.206+0000 7f9239ee2640 1 -- start start 2024-09-13T15:26:12.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.207+0000 7f9239ee2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234103dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:12.167 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.207+0000 7f9239ee2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234104390 con 0x7f92341039c0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.207+0000 7f92337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234103dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.207+0000 7f92337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234103dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47262/0 (socket says 172.21.15.90:47262) 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.207+0000 7f92337fe640 1 -- 172.21.15.90:0/1877169027 learned_addr learned my addr 172.21.15.90:0/1877169027 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.207+0000 7f92337fe640 1 -- 172.21.15.90:0/1877169027 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234104bc0 con 0x7f92341039c0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.208+0000 7f92337fe640 1 --2- 172.21.15.90:0/1877169027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234103dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9224009e80 tx=0x7f922402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3311eb75eeac29cb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.208+0000 7f92327fc640 1 -- 172.21.15.90:0/1877169027 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9224031400 con 0x7f92341039c0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.208+0000 7f92327fc640 1 -- 172.21.15.90:0/1877169027 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f922402d050 con 0x7f92341039c0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.208+0000 7f92327fc640 1 -- 172.21.15.90:0/1877169027 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9224007720 con 0x7f92341039c0 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 -- 172.21.15.90:0/1877169027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 msgr2=0x7f9234103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 --2- 172.21.15.90:0/1877169027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234103dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9224009e80 tx=0x7f922402a7a0 comp rx=0 tx=0).stop 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 -- 172.21.15.90:0/1877169027 shutdown_connections 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 --2- 172.21.15.90:0/1877169027 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234103dc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:12.168 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 -- 172.21.15.90:0/1877169027 >> 172.21.15.90:0/1877169027 conn(0x7f92340ff110 msgr2=0x7f9234101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 -- 172.21.15.90:0/1877169027 shutdown_connections 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 -- 172.21.15.90:0/1877169027 wait complete. 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 Processor -- start 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.209+0000 7f9239ee2640 1 -- start start 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9239ee2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9239ee2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234198cb0 con 0x7f92341039c0 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f92337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f92337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47278/0 (socket says 172.21.15.90:47278) 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f92337fe640 1 -- 172.21.15.90:0/3613883019 learned_addr learned my addr 172.21.15.90:0/3613883019 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f92337fe640 1 -- 172.21.15.90:0/3613883019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9224009b30 con 0x7f92341039c0 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f92337fe640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234198770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9224009e50 tx=0x7f9224031630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:12.169 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9224031a50 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9224031bb0 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9224039510 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9234198eb0 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.210+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9234199350 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.211+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7f9224038040 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.211+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92341929d0 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.211+0000 7f9230ff9640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f921c038760 0x7f921c03ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.212+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f922406bb80 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.212+0000 7f9232ffd640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f921c038760 0x7f921c03ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.212+0000 7f9232ffd640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f921c038760 0x7f921c03ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92280098c0 tx=0x7f9228006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.216+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f922404d300 con 0x7f92341039c0 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.390+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9234061900 con 0x7f921c038760 2024-09-13T15:26:12.170 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.610+0000 7f9230ff9640 1 -- 172.21.15.90:0/3613883019 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9234061900 con 0x7f921c038760 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f921c038760 msgr2=0x7f921c03ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f921c038760 0x7f921c03ac20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92280098c0 tx=0x7f9228006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 msgr2=0x7f9234198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234198770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9224009e50 tx=0x7f9224031630 comp rx=0 tx=0).stop 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 shutdown_connections 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f921c038760 0x7f921c03ac20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 --2- 172.21.15.90:0/3613883019 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f92341039c0 0x7f9234198770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.612+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 >> 172.21.15.90:0/3613883019 conn(0x7f92340ff110 msgr2=0x7f9234100030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.613+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 shutdown_connections 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:11.613+0000 7f9239ee2640 1 -- 172.21.15.90:0/3613883019 wait complete. 2024-09-13T15:26:12.171 INFO:teuthology.orchestra.run.smithi090.stdout:Fetching dashboard port number... 2024-09-13T15:26:12.724 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:12 smithi090 ceph-mon[27811]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:26:12.724 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:12 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:13.309 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stdout 8443 2024-09-13T15:26:13.309 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff254d1b640 1 Processor -- start 2024-09-13T15:26:13.309 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff254d1b640 1 -- start start 2024-09-13T15:26:13.309 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff254d1b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff2501079a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:13.309 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff254d1b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff250077e70 con 0x7ff2501055b0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff24e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff2501079a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff24e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff2501079a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47286/0 (socket says 172.21.15.90:47286) 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.547+0000 7ff24e59c640 1 -- 172.21.15.90:0/2767173938 learned_addr learned my addr 172.21.15.90:0/2767173938 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.548+0000 7ff24e59c640 1 -- 172.21.15.90:0/2767173938 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250107ee0 con 0x7ff2501055b0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.548+0000 7ff24e59c640 1 --2- 172.21.15.90:0/2767173938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff2501079a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff240009820 tx=0x7ff24002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d19abdb345d117e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.548+0000 7ff24dd9b640 1 -- 172.21.15.90:0/2767173938 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff24002ad40 con 0x7ff2501055b0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff24dd9b640 1 -- 172.21.15.90:0/2767173938 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff2400328d0 con 0x7ff2501055b0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff24dd9b640 1 -- 172.21.15.90:0/2767173938 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2400306f0 con 0x7ff2501055b0 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 -- 172.21.15.90:0/2767173938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 msgr2=0x7ff2501079a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 --2- 172.21.15.90:0/2767173938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff2501079a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff240009820 tx=0x7ff24002a270 comp rx=0 tx=0).stop 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 -- 172.21.15.90:0/2767173938 shutdown_connections 2024-09-13T15:26:13.310 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 --2- 172.21.15.90:0/2767173938 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff2501079a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 -- 172.21.15.90:0/2767173938 >> 172.21.15.90:0/2767173938 conn(0x7ff2500ff210 msgr2=0x7ff250101630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 -- 172.21.15.90:0/2767173938 shutdown_connections 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.549+0000 7ff254d1b640 1 -- 172.21.15.90:0/2767173938 wait complete. 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff254d1b640 1 Processor -- start 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff254d1b640 1 -- start start 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff254d1b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff250198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff254d1b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff250198c60 con 0x7ff2501055b0 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff24e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff250198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff24e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff250198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47288/0 (socket says 172.21.15.90:47288) 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.550+0000 7ff24e59c640 1 -- 172.21.15.90:0/1509577170 learned_addr learned my addr 172.21.15.90:0/1509577170 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff24e59c640 1 -- 172.21.15.90:0/1509577170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2400094d0 con 0x7ff2501055b0 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff24e59c640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff250198720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff240032d40 tx=0x7ff240032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff240030c10 con 0x7ff2501055b0 2024-09-13T15:26:13.311 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff240030d70 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff250198e60 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff24003a520 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.551+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff250199300 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.552+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 12) v1 ==== 45569+0+0 (secure 0 0 0) 0x7ff24003a7e0 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.552+0000 7ff247fff640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7ff23c038710 0x7ff23c03abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.552+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff24006bb90 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.552+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff250108c80 con 0x7ff2501055b0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.552+0000 7ff245d9b640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7ff23c038710 0x7ff23c03abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.552+0000 7ff245d9b640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7ff23c038710 0x7ff23c03abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff2380098c0 tx=0x7ff238006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:13.312 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.557+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7ff240043310 con 0x7ff2501055b0 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.719+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7ff250108e70 con 0x7ff2501055b0 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.720+0000 7ff247fff640 1 -- 172.21.15.90:0/1509577170 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7ff240037ed0 con 0x7ff2501055b0 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7ff23c038710 msgr2=0x7ff23c03abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7ff23c038710 0x7ff23c03abd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff2380098c0 tx=0x7ff238006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 msgr2=0x7ff250198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff250198720 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff240032d40 tx=0x7ff240032d70 comp rx=0 tx=0).stop 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 shutdown_connections 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7ff23c038710 0x7ff23c03abd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 --2- 172.21.15.90:0/1509577170 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff2501055b0 0x7ff250198720 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 >> 172.21.15.90:0/1509577170 conn(0x7ff2500ff210 msgr2=0x7ff2500ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 shutdown_connections 2024-09-13T15:26:13.313 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:12.722+0000 7ff254d1b640 1 -- 172.21.15.90:0/1509577170 wait complete. 2024-09-13T15:26:13.316 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-13T15:26:13.316 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout disabled 2024-09-13T15:26:13.324 INFO:teuthology.orchestra.run.smithi090.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-13T15:26:13.324 INFO:teuthology.orchestra.run.smithi090.stdout:systemctl: stdout inactive 2024-09-13T15:26:13.324 INFO:teuthology.orchestra.run.smithi090.stdout:firewalld.service is not enabled 2024-09-13T15:26:13.324 INFO:teuthology.orchestra.run.smithi090.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout:Ceph Dashboard is now available at: 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout: URL: https://smithi090.front.sepia.ceph.com:8443/ 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout: User: admin 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout: Password: v53cbjc2y1 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:13.326 INFO:teuthology.orchestra.run.smithi090.stdout:Saving cluster configuration to /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config directory 2024-09-13T15:26:13.327 INFO:teuthology.orchestra.run.smithi090.stdout:Enabling autotune for osd_memory_target 2024-09-13T15:26:13.887 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:13 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1509577170' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-13T15:26:14.438 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.714+0000 7fc891769640 1 Processor -- start 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.714+0000 7fc891769640 1 -- start start 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.714+0000 7fc891769640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c1037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.714+0000 7fc891769640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c077e50 con 0x7fc88c1033b0 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.715+0000 7fc88affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c1037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.715+0000 7fc88affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c1037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47304/0 (socket says 172.21.15.90:47304) 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.715+0000 7fc88affd640 1 -- 172.21.15.90:0/1651479734 learned_addr learned my addr 172.21.15.90:0/1651479734 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.715+0000 7fc88affd640 1 -- 172.21.15.90:0/1651479734 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c103cf0 con 0x7fc88c1033b0 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc88affd640 1 --2- 172.21.15.90:0/1651479734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c1037b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009820 tx=0x7fc87c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15c144342fed712 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc889ffb640 1 -- 172.21.15.90:0/1651479734 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c02ad40 con 0x7fc88c1033b0 2024-09-13T15:26:14.443 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc889ffb640 1 -- 172.21.15.90:0/1651479734 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc87c0328d0 con 0x7fc88c1033b0 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc889ffb640 1 -- 172.21.15.90:0/1651479734 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c0306f0 con 0x7fc88c1033b0 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 -- 172.21.15.90:0/1651479734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 msgr2=0x7fc88c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 --2- 172.21.15.90:0/1651479734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c1037b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc87c009820 tx=0x7fc87c02a270 comp rx=0 tx=0).stop 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 -- 172.21.15.90:0/1651479734 shutdown_connections 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 --2- 172.21.15.90:0/1651479734 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c1037b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 -- 172.21.15.90:0/1651479734 >> 172.21.15.90:0/1651479734 conn(0x7fc88c0ff180 msgr2=0x7fc88c1015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 -- 172.21.15.90:0/1651479734 shutdown_connections 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.716+0000 7fc891769640 1 -- 172.21.15.90:0/1651479734 wait complete. 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc891769640 1 Processor -- start 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc891769640 1 -- start start 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc891769640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc891769640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c19ae70 con 0x7fc88c1033b0 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc88affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:14.444 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc88affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c19a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47312/0 (socket says 172.21.15.90:47312) 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.717+0000 7fc88affd640 1 -- 172.21.15.90:0/4239145261 learned_addr learned my addr 172.21.15.90:0/4239145261 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc88affd640 1 -- 172.21.15.90:0/4239145261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c0094d0 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc88affd640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c19a930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc87c032d40 tx=0x7fc87c032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c030c10 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc87c030d70 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c19b070 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87c03a520 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.718+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c19b510 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.719+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fc87c03a7e0 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.719+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c104bb0 con 0x7fc88c1033b0 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.719+0000 7fc883fff640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc8780387b0 0x7fc87803ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:14.445 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.719+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc87c06bdb0 con 0x7fc88c1033b0 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.719+0000 7fc88a7fc640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc8780387b0 0x7fc87803ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.720+0000 7fc88a7fc640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc8780387b0 0x7fc87803ac70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8740098c0 tx=0x7fc874006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.724+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fc87c043310 con 0x7fc88c1033b0 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.884+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fc88c104df0 con 0x7fc88c1033b0 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.885+0000 7fc883fff640 1 -- 172.21.15.90:0/4239145261 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fc87c045e40 con 0x7fc88c1033b0 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc8780387b0 msgr2=0x7fc87803ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc8780387b0 0x7fc87803ac70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8740098c0 tx=0x7fc874006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 msgr2=0x7fc88c19a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c19a930 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc87c032d40 tx=0x7fc87c032d70 comp rx=0 tx=0).stop 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 shutdown_connections 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc8780387b0 0x7fc87803ac70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:14.446 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 --2- 172.21.15.90:0/4239145261 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc88c1033b0 0x7fc88c19a930 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:14.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 >> 172.21.15.90:0/4239145261 conn(0x7fc88c0ff180 msgr2=0x7fc88c18efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:14.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 shutdown_connections 2024-09-13T15:26:14.447 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:13.887+0000 7fc891769640 1 -- 172.21.15.90:0/4239145261 wait complete. 2024-09-13T15:26:14.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:14 smithi090 ceph-mon[27811]: mgrmap e13: smithi090.fyvvpd(active, since 6s) 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.840+0000 7fae61f6a640 1 Processor -- start 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae61f6a640 1 -- start start 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae61f6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c107920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae61f6a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c077e70 con 0x7fae5c105530 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae5b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c107920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae5b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c107920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56572/0 (socket says 172.21.15.90:56572) 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae5b7fe640 1 -- 172.21.15.90:0/2124889730 learned_addr learned my addr 172.21.15.90:0/2124889730 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:15.660 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.841+0000 7fae5b7fe640 1 -- 172.21.15.90:0/2124889730 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c107e60 con 0x7fae5c105530 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.842+0000 7fae5b7fe640 1 --2- 172.21.15.90:0/2124889730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c107920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009820 tx=0x7fae4c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cd78fabe50c2f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.842+0000 7fae5affd640 1 -- 172.21.15.90:0/2124889730 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4c02ad40 con 0x7fae5c105530 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.842+0000 7fae5affd640 1 -- 172.21.15.90:0/2124889730 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fae4c0328d0 con 0x7fae5c105530 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.842+0000 7fae5affd640 1 -- 172.21.15.90:0/2124889730 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4c0306f0 con 0x7fae5c105530 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- 172.21.15.90:0/2124889730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 msgr2=0x7fae5c107920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 --2- 172.21.15.90:0/2124889730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c107920 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009820 tx=0x7fae4c02a270 comp rx=0 tx=0).stop 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- 172.21.15.90:0/2124889730 shutdown_connections 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 --2- 172.21.15.90:0/2124889730 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c107920 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- 172.21.15.90:0/2124889730 >> 172.21.15.90:0/2124889730 conn(0x7fae5c0ff130 msgr2=0x7fae5c101550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- 172.21.15.90:0/2124889730 shutdown_connections 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- 172.21.15.90:0/2124889730 wait complete. 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 Processor -- start 2024-09-13T15:26:15.661 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- start start 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c104db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.843+0000 7fae61f6a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c1052f0 con 0x7fae5c105530 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae5b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c104db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae5b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c104db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56586/0 (socket says 172.21.15.90:56586) 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae5b7fe640 1 -- 172.21.15.90:0/1094172862 learned_addr learned my addr 172.21.15.90:0/1094172862 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae5b7fe640 1 -- 172.21.15.90:0/1094172862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4c0094d0 con 0x7fae5c105530 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae5b7fe640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c104db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fae4c032d40 tx=0x7fae4c032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4c030c10 con 0x7fae5c105530 2024-09-13T15:26:15.662 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fae4c030d70 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4c03a520 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c1034e0 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.844+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c103980 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.845+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fae4c03c400 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.845+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c108c00 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.845+0000 7fae597fa640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fae48038800 0x7fae4803acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.846+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fae4c06bbb0 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.846+0000 7fae53fff640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fae48038800 0x7fae4803acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.846+0000 7fae53fff640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fae48038800 0x7fae4803acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae440098c0 tx=0x7fae44006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:14.850+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fae4c0434f0 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.098+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fae5c108df0 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.101+0000 7fae597fa640 1 -- 172.21.15.90:0/1094172862 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v34)=0 set mgr/dashboard/cluster/status v34) v1 ==== 153+0+0 (secure 0 0 0) 0x7fae4c043700 con 0x7fae5c105530 2024-09-13T15:26:15.663 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fae48038800 msgr2=0x7fae4803acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fae48038800 0x7fae4803acc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fae440098c0 tx=0x7fae44006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 msgr2=0x7fae5c104db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c104db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fae4c032d40 tx=0x7fae4c032d70 comp rx=0 tx=0).stop 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 shutdown_connections 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fae48038800 0x7fae4803acc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 --2- 172.21.15.90:0/1094172862 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae5c105530 0x7fae5c104db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.103+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 >> 172.21.15.90:0/1094172862 conn(0x7fae5c0ff130 msgr2=0x7fae5c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.104+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 shutdown_connections 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:/usr/bin/ceph: stderr 2024-09-13T15:26:15.104+0000 7fae61f6a640 1 -- 172.21.15.90:0/1094172862 wait complete. 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.664 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout:Or, if you are only running a single cluster on this host: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: ceph telemetry on 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout:For more information see: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.665 INFO:teuthology.orchestra.run.smithi090.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-13T15:26:15.666 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:15.666 INFO:teuthology.orchestra.run.smithi090.stdout:Bootstrap complete. 2024-09-13T15:26:15.692 INFO:tasks.cephadm:Fetching config... 2024-09-13T15:26:15.692 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:26:15.692 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-13T15:26:15.713 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-13T15:26:15.713 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:26:15.713 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-13T15:26:15.769 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-13T15:26:15.769 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:26:15.769 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/keyring of=/dev/stdout 2024-09-13T15:26:15.834 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-13T15:26:15.834 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:26:15.834 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-13T15:26:15.890 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-13T15:26:15.890 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqIQdihJcO89NCks6rs+mLqYXSGlM24HqpneJm5+KqQv66tEGe/2YujLyHNTuXLd9nAgi3ufYAikcmdFOXAilUQYwwQ70o7XbKgFQsLt3iCT5S9uoBO6Y0WhNan3Jvlteqw8/vmZwuyDwOU2iQ5YgTkU+yRywhD/OIkXAqltYOMvOCdRoUbuch129NjpuWLbGKaA7nKOztpEMEY2Odtx39Nvk0ii3qivABnI2CpDbKE1Hngje3sfng7etewm4cOSBFDDvWuhk9ny6zoF6w7ah9UHi9KPUV2q89ioYwOEsbb+/riJK51RZRkqPgYgS1A4jV48Pvwz0znM6Gjh7TJ/FNvocmxgrZ6/k0MnbTxJ8k8RX/qDpkDJCOoCcJrRDH7qNbvQAHDpyD87SUcMx05k6muIZBmn9WNXSKwPtc1SpnRzSYxUSBqKPf1vvhY4Hx+6LEz6naBjs15sWMSNTRhzG+YL6Vd6N8Sq5gDIHBYgSOdBE+lsSPxRvQ3uRnt/pHSUk= ceph-41124e1c-71e4-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T15:26:15.968 INFO:teuthology.orchestra.run.smithi090.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqIQdihJcO89NCks6rs+mLqYXSGlM24HqpneJm5+KqQv66tEGe/2YujLyHNTuXLd9nAgi3ufYAikcmdFOXAilUQYwwQ70o7XbKgFQsLt3iCT5S9uoBO6Y0WhNan3Jvlteqw8/vmZwuyDwOU2iQ5YgTkU+yRywhD/OIkXAqltYOMvOCdRoUbuch129NjpuWLbGKaA7nKOztpEMEY2Odtx39Nvk0ii3qivABnI2CpDbKE1Hngje3sfng7etewm4cOSBFDDvWuhk9ny6zoF6w7ah9UHi9KPUV2q89ioYwOEsbb+/riJK51RZRkqPgYgS1A4jV48Pvwz0znM6Gjh7TJ/FNvocmxgrZ6/k0MnbTxJ8k8RX/qDpkDJCOoCcJrRDH7qNbvQAHDpyD87SUcMx05k6muIZBmn9WNXSKwPtc1SpnRzSYxUSBqKPf1vvhY4Hx+6LEz6naBjs15sWMSNTRhzG+YL6Vd6N8Sq5gDIHBYgSOdBE+lsSPxRvQ3uRnt/pHSUk= ceph-41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:26:15.980 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqIQdihJcO89NCks6rs+mLqYXSGlM24HqpneJm5+KqQv66tEGe/2YujLyHNTuXLd9nAgi3ufYAikcmdFOXAilUQYwwQ70o7XbKgFQsLt3iCT5S9uoBO6Y0WhNan3Jvlteqw8/vmZwuyDwOU2iQ5YgTkU+yRywhD/OIkXAqltYOMvOCdRoUbuch129NjpuWLbGKaA7nKOztpEMEY2Odtx39Nvk0ii3qivABnI2CpDbKE1Hngje3sfng7etewm4cOSBFDDvWuhk9ny6zoF6w7ah9UHi9KPUV2q89ioYwOEsbb+/riJK51RZRkqPgYgS1A4jV48Pvwz0znM6Gjh7TJ/FNvocmxgrZ6/k0MnbTxJ8k8RX/qDpkDJCOoCcJrRDH7qNbvQAHDpyD87SUcMx05k6muIZBmn9WNXSKwPtc1SpnRzSYxUSBqKPf1vvhY4Hx+6LEz6naBjs15sWMSNTRhzG+YL6Vd6N8Sq5gDIHBYgSOdBE+lsSPxRvQ3uRnt/pHSUk= ceph-41124e1c-71e4-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-13T15:26:16.036 INFO:teuthology.orchestra.run.smithi175.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqIQdihJcO89NCks6rs+mLqYXSGlM24HqpneJm5+KqQv66tEGe/2YujLyHNTuXLd9nAgi3ufYAikcmdFOXAilUQYwwQ70o7XbKgFQsLt3iCT5S9uoBO6Y0WhNan3Jvlteqw8/vmZwuyDwOU2iQ5YgTkU+yRywhD/OIkXAqltYOMvOCdRoUbuch129NjpuWLbGKaA7nKOztpEMEY2Odtx39Nvk0ii3qivABnI2CpDbKE1Hngje3sfng7etewm4cOSBFDDvWuhk9ny6zoF6w7ah9UHi9KPUV2q89ioYwOEsbb+/riJK51RZRkqPgYgS1A4jV48Pvwz0znM6Gjh7TJ/FNvocmxgrZ6/k0MnbTxJ8k8RX/qDpkDJCOoCcJrRDH7qNbvQAHDpyD87SUcMx05k6muIZBmn9WNXSKwPtc1SpnRzSYxUSBqKPf1vvhY4Hx+6LEz6naBjs15sWMSNTRhzG+YL6Vd6N8Sq5gDIHBYgSOdBE+lsSPxRvQ3uRnt/pHSUk= ceph-41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:26:16.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-13T15:26:16.242 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:26:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:16 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1094172862' entity='client.admin' 2024-09-13T15:26:17.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.896+0000 7f3080263640 1 -- 172.21.15.90:0/1159671119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 msgr2=0x7f30780fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:17.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.896+0000 7f3080263640 1 --2- 172.21.15.90:0/1159671119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fead0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3070009880 tx=0x7f307002a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 -- 172.21.15.90:0/1159671119 shutdown_connections 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 --2- 172.21.15.90:0/1159671119 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fead0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 -- 172.21.15.90:0/1159671119 >> 172.21.15.90:0/1159671119 conn(0x7f30780f9e80 msgr2=0x7f30780fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 -- 172.21.15.90:0/1159671119 shutdown_connections 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 -- 172.21.15.90:0/1159671119 wait complete. 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 Processor -- start 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.897+0000 7f3080263640 1 -- start start 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f3080263640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f3080263640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30780fc230 con 0x7f30780fe6d0 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f307dfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f307dfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56596/0 (socket says 172.21.15.90:56596) 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f307dfff640 1 -- 172.21.15.90:0/4200942949 learned_addr learned my addr 172.21.15.90:0/4200942949 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:17.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f307dfff640 1 -- 172.21.15.90:0/4200942949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3070009560 con 0x7f30780fe6d0 2024-09-13T15:26:17.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f307dfff640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fbcf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3070009230 tx=0x7f30700051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30700055c0 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3070003a80 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30780fa420 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.898+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307003ca20 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.899+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30780fa8c0 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.899+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f307003cb80 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.900+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3078104c60 con 0x7f30780fe6d0 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.900+0000 7f3076ffd640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f3060038740 0x7f306003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:17.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.900+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f307006c650 con 0x7f30780fe6d0 2024-09-13T15:26:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.900+0000 7f307d7fe640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f3060038740 0x7f306003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:17.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.900+0000 7f307d7fe640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f3060038740 0x7f306003ac00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f306c0098c0 tx=0x7f306c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:17.906 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:17.904+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f307004dbd0 con 0x7f30780fe6d0 2024-09-13T15:26:18.079 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.077+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f30780feb50 con 0x7f30780fe6d0 2024-09-13T15:26:18.082 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.080+0000 7f3076ffd640 1 -- 172.21.15.90:0/4200942949 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f30700350c0 con 0x7f30780fe6d0 2024-09-13T15:26:18.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f3060038740 msgr2=0x7f306003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f3060038740 0x7f306003ac00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f306c0098c0 tx=0x7f306c006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 msgr2=0x7f30780fbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fbcf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3070009230 tx=0x7f30700051a0 comp rx=0 tx=0).stop 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 shutdown_connections 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f3060038740 0x7f306003ac00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 --2- 172.21.15.90:0/4200942949 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f30780fe6d0 0x7f30780fbcf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 >> 172.21.15.90:0/4200942949 conn(0x7f30780f9e80 msgr2=0x7f3078067f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 shutdown_connections 2024-09-13T15:26:18.085 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:18.082+0000 7f3080263640 1 -- 172.21.15.90:0/4200942949 wait complete. 2024-09-13T15:26:18.835 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-13T15:26:18.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-13T15:26:19.255 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/4200942949' entity='client.admin' 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:26:19.377 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:19 smithi090 ceph-mon[27811]: Deploying daemon ceph-exporter.smithi090 on smithi090 2024-09-13T15:26:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.476+0000 7f6bb0f71640 1 -- 172.21.15.90:0/2364982517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 msgr2=0x7f6bac0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.476+0000 7f6bb0f71640 1 --2- 172.21.15.90:0/2364982517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac0fe4b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009c20 tx=0x7f6b9c02aa90 comp rx=0 tx=0).stop 2024-09-13T15:26:21.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 -- 172.21.15.90:0/2364982517 shutdown_connections 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 --2- 172.21.15.90:0/2364982517 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac0fe4b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 -- 172.21.15.90:0/2364982517 >> 172.21.15.90:0/2364982517 conn(0x7f6bac0f9e80 msgr2=0x7f6bac0fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 -- 172.21.15.90:0/2364982517 shutdown_connections 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 -- 172.21.15.90:0/2364982517 wait complete. 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 Processor -- start 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.477+0000 7f6bb0f71640 1 -- start start 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6bb0f71640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6bb0f71640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac197140 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6baa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6baa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac196c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56614/0 (socket says 172.21.15.90:56614) 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6baa59c640 1 -- 172.21.15.90:0/975319195 learned_addr learned my addr 172.21.15.90:0/975319195 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6baa59c640 1 -- 172.21.15.90:0/975319195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c009900 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6baa59c640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac196c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c02e5f0 tx=0x7f6b9c02e620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9c03e070 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac197340 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6b9c02fb00 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b9c004820 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.478+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac1977e0 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.479+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f6b9c00a040 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.479+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac09f180 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.479+0000 7f6b9b7fe640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6b94038740 0x7f6b9403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.479+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b9c06c850 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.479+0000 7f6ba9d9b640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6b94038740 0x7f6b9403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:21.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.480+0000 7f6ba9d9b640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6b94038740 0x7f6b9403ac00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098c0 tx=0x7f6ba0006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.484+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6b9c0425b0 con 0x7f6bac0fe0b0 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.656+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bac0608f0 con 0x7f6b94038740 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.660+0000 7f6b9b7fe640 1 -- 172.21.15.90:0/975319195 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6bac0608f0 con 0x7f6b94038740 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6b94038740 msgr2=0x7f6b9403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6b94038740 0x7f6b9403ac00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098c0 tx=0x7f6ba0006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 msgr2=0x7f6bac196c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac196c00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c02e5f0 tx=0x7f6b9c02e620 comp rx=0 tx=0).stop 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 shutdown_connections 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6b94038740 0x7f6b9403ac00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 --2- 172.21.15.90:0/975319195 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6bac0fe0b0 0x7f6bac196c00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 >> 172.21.15.90:0/975319195 conn(0x7f6bac0f9e80 msgr2=0x7f6bac0fa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:21.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 shutdown_connections 2024-09-13T15:26:21.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:21.662+0000 7f6bb0f71640 1 -- 172.21.15.90:0/975319195 wait complete. 2024-09-13T15:26:22.383 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi175 2024-09-13T15:26:22.383 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:26:22.383 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/etc/ceph/ceph.conf 2024-09-13T15:26:22.402 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:26:22.402 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T15:26:22.462 INFO:tasks.cephadm:Adding host smithi175 to orchestrator... 2024-09-13T15:26:22.462 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch host add smithi175 2024-09-13T15:26:22.886 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:26:23.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:22 smithi090 ceph-mon[27811]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:26:23.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:22 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 -- 172.21.15.90:0/846034379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe80fe6f0 msgr2=0x7fdfe80feaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 --2- 172.21.15.90:0/846034379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe80fe6f0 0x7fdfe80feaf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009880 tx=0x7fdfd802a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 -- 172.21.15.90:0/846034379 shutdown_connections 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 --2- 172.21.15.90:0/846034379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe80fe6f0 0x7fdfe80feaf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 -- 172.21.15.90:0/846034379 >> 172.21.15.90:0/846034379 conn(0x7fdfe80f9e80 msgr2=0x7fdfe80fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 -- 172.21.15.90:0/846034379 shutdown_connections 2024-09-13T15:26:24.917 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.915+0000 7fdfedc7b640 1 -- 172.21.15.90:0/846034379 wait complete. 2024-09-13T15:26:24.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfedc7b640 1 Processor -- start 2024-09-13T15:26:24.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfedc7b640 1 -- start start 2024-09-13T15:26:24.919 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfedc7b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 0x7fdfe81951a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfedc7b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe81956e0 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 0x7fdfe81951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfe77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 0x7fdfe81951a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47984/0 (socket says 172.21.15.90:47984) 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfe77fe640 1 -- 172.21.15.90:0/4244107654 learned_addr learned my addr 172.21.15.90:0/4244107654 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfe77fe640 1 -- 172.21.15.90:0/4244107654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd8009560 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.916+0000 7fdfe77fe640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 0x7fdfe81951a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009230 tx=0x7fdfd8004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.917+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd8005460 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.917+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe81958e0 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.917+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdfd80055c0 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.917+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd803c850 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.917+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe8198450 con 0x7fdfe8194d80 2024-09-13T15:26:24.920 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.917+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fdfd803c9b0 con 0x7fdfe8194d80 2024-09-13T15:26:24.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.918+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe80feaf0 con 0x7fdfe8194d80 2024-09-13T15:26:24.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.918+0000 7fdfe57fa640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdfd4038790 0x7fdfd403ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:24.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.918+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdfd806c2a0 con 0x7fdfe8194d80 2024-09-13T15:26:24.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.918+0000 7fdfdffff640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdfd4038790 0x7fdfd403ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:24.921 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.918+0000 7fdfdffff640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdfd4038790 0x7fdfd403ac50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00098c0 tx=0x7fdfd0006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:24.923 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:24.922+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fdfd8031540 con 0x7fdfe8194d80 2024-09-13T15:26:25.099 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:25.097+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi175", "target": ["mon-mgr", ""]}) v1 -- 0x7fdfe80685d0 con 0x7fdfd4038790 2024-09-13T15:26:25.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:25.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: Deploying daemon crash.smithi090 on smithi090 2024-09-13T15:26:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:25 smithi090 ceph-mon[27811]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi175", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:26:26.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:26 smithi090 ceph-mon[27811]: Deploying cephadm binary to smithi175 2024-09-13T15:26:27.822 INFO:teuthology.orchestra.run.smithi090.stdout:Added host 'smithi175' with addr '172.21.15.175' 2024-09-13T15:26:27.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.819+0000 7fdfe57fa640 1 -- 172.21.15.90:0/4244107654 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdfe80685d0 con 0x7fdfd4038790 2024-09-13T15:26:27.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdfd4038790 msgr2=0x7fdfd403ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:27.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdfd4038790 0x7fdfd403ac50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00098c0 tx=0x7fdfd0006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 msgr2=0x7fdfe81951a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 0x7fdfe81951a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009230 tx=0x7fdfd8004f90 comp rx=0 tx=0).stop 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 shutdown_connections 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdfd4038790 0x7fdfd403ac50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 --2- 172.21.15.90:0/4244107654 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfe8194d80 0x7fdfe81951a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 >> 172.21.15.90:0/4244107654 conn(0x7fdfe80f9e80 msgr2=0x7fdfe8067ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 shutdown_connections 2024-09-13T15:26:27.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:27.822+0000 7fdfedc7b640 1 -- 172.21.15.90:0/4244107654 wait complete. 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: Deploying daemon node-exporter.smithi090 on smithi090 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:28.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:27 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:28.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-13T15:26:28.623 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:26:29.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:28 smithi090 ceph-mon[27811]: Added host smithi175 2024-09-13T15:26:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:29 smithi090 ceph-mon[27811]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:31.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 -- 172.21.15.90:0/3916125772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 msgr2=0x7f1c3c0ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:31.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 --2- 172.21.15.90:0/3916125772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c0ffbe0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009880 tx=0x7f1c2c02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:31.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 -- 172.21.15.90:0/3916125772 shutdown_connections 2024-09-13T15:26:31.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 --2- 172.21.15.90:0/3916125772 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c0ffbe0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 -- 172.21.15.90:0/3916125772 >> 172.21.15.90:0/3916125772 conn(0x7f1c3c0faf90 msgr2=0x7f1c3c0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 -- 172.21.15.90:0/3916125772 shutdown_connections 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.318+0000 7f1c415bf640 1 -- 172.21.15.90:0/3916125772 wait complete. 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.319+0000 7f1c415bf640 1 Processor -- start 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.319+0000 7f1c415bf640 1 -- start start 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.319+0000 7f1c415bf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c194c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.319+0000 7f1c415bf640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c3c195140 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c3affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c194c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:31.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c3affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c194c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48016/0 (socket says 172.21.15.90:48016) 2024-09-13T15:26:31.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c3affd640 1 -- 172.21.15.90:0/948151117 learned_addr learned my addr 172.21.15.90:0/948151117 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:31.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c3affd640 1 -- 172.21.15.90:0/948151117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c009560 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c3affd640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c194c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009230 tx=0x7f1c2c0051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c2c0035f0 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1c2c003be0 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c2c03c980 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3c195340 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.320+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c3c1957e0 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.321+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f1c2c03cb70 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.321+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c3c0ffc60 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.321+0000 7f1c23fff640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f1c24038740 0x7f1c2403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.321+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1c2c06c590 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.321+0000 7f1c3a7fc640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f1c24038740 0x7f1c2403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.322+0000 7f1c3a7fc640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f1c24038740 0x7f1c2403ac00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1c300098c0 tx=0x7f1c30006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:31.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.326+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1c2c03abe0 con 0x7f1c3c0ff7e0 2024-09-13T15:26:31.496 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.495+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c3c0608f0 con 0x7f1c24038740 2024-09-13T15:26:31.498 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.497+0000 7f1c23fff640 1 -- 172.21.15.90:0/948151117 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f1c3c0608f0 con 0x7f1c24038740 2024-09-13T15:26:31.498 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:26:31.498 INFO:teuthology.orchestra.run.smithi090.stdout:[{"addr": "172.21.15.90", "hostname": "smithi090", "labels": [], "status": ""}, {"addr": "172.21.15.175", "hostname": "smithi175", "labels": [], "status": ""}] 2024-09-13T15:26:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f1c24038740 msgr2=0x7f1c2403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f1c24038740 0x7f1c2403ac00 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1c300098c0 tx=0x7f1c30006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 msgr2=0x7f1c3c194c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c194c00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1c2c009230 tx=0x7f1c2c0051a0 comp rx=0 tx=0).stop 2024-09-13T15:26:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 shutdown_connections 2024-09-13T15:26:31.500 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f1c24038740 0x7f1c2403ac00 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:31.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 --2- 172.21.15.90:0/948151117 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1c3c0ff7e0 0x7f1c3c194c00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:31.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.498+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 >> 172.21.15.90:0/948151117 conn(0x7f1c3c0faf90 msgr2=0x7f1c3c0fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:31.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.499+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 shutdown_connections 2024-09-13T15:26:31.501 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:31.499+0000 7f1c415bf640 1 -- 172.21.15.90:0/948151117 wait complete. 2024-09-13T15:26:32.152 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:32 smithi090 ceph-mon[27811]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:32.154 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-13T15:26:32.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-13T15:26:32.550 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:26:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:33 smithi090 ceph-mon[27811]: from='client.14191 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:26:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:33 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:33 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:33 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:33 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:34.334 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 -- 172.21.15.90:0/4017206930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 msgr2=0x7fd3e40feae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 --2- 172.21.15.90:0/4017206930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e40feae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009880 tx=0x7fd3d402a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 -- 172.21.15.90:0/4017206930 shutdown_connections 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 --2- 172.21.15.90:0/4017206930 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e40feae0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 -- 172.21.15.90:0/4017206930 >> 172.21.15.90:0/4017206930 conn(0x7fd3e40f9e90 msgr2=0x7fd3e40fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 -- 172.21.15.90:0/4017206930 shutdown_connections 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.058+0000 7fd3e9237640 1 -- 172.21.15.90:0/4017206930 wait complete. 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e9237640 1 Processor -- start 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e9237640 1 -- start start 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e9237640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e4068940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e9237640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e4068e80 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e4068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e4068940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48042/0 (socket says 172.21.15.90:48042) 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e2d9d640 1 -- 172.21.15.90:0/2910080324 learned_addr learned my addr 172.21.15.90:0/2910080324 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.059+0000 7fd3e2d9d640 1 -- 172.21.15.90:0/2910080324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d4009560 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.060+0000 7fd3e2d9d640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e4068940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009230 tx=0x7fd3d40051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.060+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3d40055c0 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.060+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e4197930 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.060+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd3d4003a80 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.060+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3d403c820 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.060+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e4197dd0 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.061+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fd3d403caa0 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.061+0000 7fd3dbfff640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd3d0038740 0x7fd3d003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.061+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd3d406d1e0 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.061+0000 7fd3e259c640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd3d0038740 0x7fd3d003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.061+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e40feb60 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.061+0000 7fd3e259c640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd3d0038740 0x7fd3d003ac00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0098c0 tx=0x7fd3cc006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.066+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd3d4035080 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:34.235+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fd3e418d2b0 con 0x7fd3e40fe6e0 2024-09-13T15:26:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:34 smithi090 ceph-mon[27811]: Deploying daemon alertmanager.smithi090 on smithi090 2024-09-13T15:26:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:34 smithi090 ceph-mon[27811]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:35.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.123+0000 7fd3dbfff640 1 -- 172.21.15.90:0/2910080324 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd3d4040020 con 0x7fd3e40fe6e0 2024-09-13T15:26:35.124 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd3d0038740 msgr2=0x7fd3d003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd3d0038740 0x7fd3d003ac00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc0098c0 tx=0x7fd3cc006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 msgr2=0x7fd3e4068940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e4068940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009230 tx=0x7fd3d40051a0 comp rx=0 tx=0).stop 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 shutdown_connections 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fd3d0038740 0x7fd3d003ac00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 --2- 172.21.15.90:0/2910080324 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3e40fe6e0 0x7fd3e4068940 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.125+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 >> 172.21.15.90:0/2910080324 conn(0x7fd3e40f9e90 msgr2=0x7fd3e40fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.126+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 shutdown_connections 2024-09-13T15:26:35.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:26:35.126+0000 7fd3e9237640 1 -- 172.21.15.90:0/2910080324 wait complete. 2024-09-13T15:26:35.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:35 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2910080324' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-13T15:26:36.096 INFO:tasks.cephadm:Adding mon.smithi090 on smithi090 2024-09-13T15:26:36.096 INFO:tasks.cephadm:Adding mon.smithi175 on smithi175 2024-09-13T15:26:36.096 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch apply mon '2;smithi090:172.21.15.90=smithi090;smithi175:172.21.15.175=smithi175' 2024-09-13T15:26:36.293 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:36.335 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:36 smithi090 ceph-mon[27811]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:36 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2910080324' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-13T15:26:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:36 smithi090 ceph-mon[27811]: osdmap e4: 0 total, 0 up, 0 in 2024-09-13T15:26:37.850 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.845+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1446495268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 msgr2=0x7f9f880feaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:37.850 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.845+0000 7f9f8d30a640 1 --2- 172.21.15.175:0/1446495268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f880feaf0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009880 tx=0x7f9f7802a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1446495268 shutdown_connections 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 --2- 172.21.15.175:0/1446495268 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f880feaf0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1446495268 >> 172.21.15.175:0/1446495268 conn(0x7f9f880f9e80 msgr2=0x7f9f880fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1446495268 shutdown_connections 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1446495268 wait complete. 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 Processor -- start 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 -- start start 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f88194bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:37.851 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.846+0000 7f9f8d30a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f88195100 con 0x7f9f880fe6f0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f86ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f88194bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f86ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f88194bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:58652/0 (socket says 172.21.15.175:58652) 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f86ffd640 1 -- 172.21.15.175:0/1147750500 learned_addr learned my addr 172.21.15.175:0/1147750500 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f86ffd640 1 -- 172.21.15.175:0/1147750500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f78009560 con 0x7f9f880fe6f0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f86ffd640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f88194bc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009230 tx=0x7f9f780051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f780035f0 con 0x7f9f880fe6f0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f88195300 con 0x7f9f880fe6f0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9f78003be0 con 0x7f9f880fe6f0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f881957a0 con 0x7f9f880fe6f0 2024-09-13T15:26:37.852 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.847+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7803c850 con 0x7f9f880fe6f0 2024-09-13T15:26:37.853 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.848+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f9f7803c9b0 con 0x7f9f880fe6f0 2024-09-13T15:26:37.853 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.848+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f880feb70 con 0x7f9f880fe6f0 2024-09-13T15:26:37.853 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.848+0000 7f9f6ffff640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f9f70038740 0x7f9f7003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:37.853 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.848+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9f7806c6f0 con 0x7f9f880fe6f0 2024-09-13T15:26:37.853 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.849+0000 7f9f867fc640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f9f70038740 0x7f9f7003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:37.853 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.849+0000 7f9f867fc640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f9f70038740 0x7f9f7003ac00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0098c0 tx=0x7f9f7c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:37.854 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:37.853+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f9f78043360 con 0x7f9f880fe6f0 2024-09-13T15:26:38.027 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.026+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 --> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi090:172.21.15.90=smithi090;smithi175:172.21.15.175=smithi175", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f880609a0 con 0x7f9f70038740 2024-09-13T15:26:38.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.032+0000 7f9f6ffff640 1 -- 172.21.15.175:0/1147750500 <== mgr.14162 v2:172.21.15.90:6800/3319538530 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9f880609a0 con 0x7f9f70038740 2024-09-13T15:26:38.033 INFO:teuthology.orchestra.run.smithi175.stdout:Scheduled mon update... 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f9f70038740 msgr2=0x7f9f7003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f9f70038740 0x7f9f7003ac00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0098c0 tx=0x7f9f7c006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 msgr2=0x7f9f88194bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f88194bc0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9f78009230 tx=0x7f9f780051a0 comp rx=0 tx=0).stop 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 shutdown_connections 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f9f70038740 0x7f9f7003ac00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 --2- 172.21.15.175:0/1147750500 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9f880fe6f0 0x7f9f88194bc0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.033+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 >> 172.21.15.175:0/1147750500 conn(0x7f9f880f9e80 msgr2=0x7f9f880686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.034+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 shutdown_connections 2024-09-13T15:26:38.035 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:38.034+0000 7f9f8d30a640 1 -- 172.21.15.175:0/1147750500 wait complete. 2024-09-13T15:26:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:38 smithi090 ceph-mon[27811]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:38 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:38 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:38.632 DEBUG:teuthology.orchestra.run.smithi175:mon.smithi175> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi175.service 2024-09-13T15:26:38.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:38.635 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:38.854 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:38.897 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:39.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:39 smithi090 ceph-mon[27811]: from='client.14195 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi090:172.21.15.90=smithi090;smithi175:172.21.15.175=smithi175", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:26:39.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:39 smithi090 ceph-mon[27811]: Saving service mon spec with placement smithi090:172.21.15.90=smithi090;smithi175:172.21.15.175=smithi175;count:2 2024-09-13T15:26:40.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 -- 172.21.15.175:0/4033255240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 msgr2=0x7fa2440fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 --2- 172.21.15.175:0/4033255240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa2440fead0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009880 tx=0x7fa22c02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 -- 172.21.15.175:0/4033255240 shutdown_connections 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 --2- 172.21.15.175:0/4033255240 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa2440fead0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 -- 172.21.15.175:0/4033255240 >> 172.21.15.175:0/4033255240 conn(0x7fa2440f9e80 msgr2=0x7fa2440fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 -- 172.21.15.175:0/4033255240 shutdown_connections 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.372+0000 7fa24ad39640 1 -- 172.21.15.175:0/4033255240 wait complete. 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa24ad39640 1 Processor -- start 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa24ad39640 1 -- start start 2024-09-13T15:26:40.376 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa24ad39640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa244194c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa24ad39640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2441951a0 con 0x7fa2440fe6d0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa248ad5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa244194c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa248ad5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa244194c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:58904/0 (socket says 172.21.15.175:58904) 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.373+0000 7fa248ad5640 1 -- 172.21.15.175:0/3824509972 learned_addr learned my addr 172.21.15.175:0/3824509972 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa248ad5640 1 -- 172.21.15.175:0/3824509972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22c009560 con 0x7fa2440fe6d0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa248ad5640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa244194c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009230 tx=0x7fa22c004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22c0055a0 con 0x7fa2440fe6d0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa22c003a80 con 0x7fa2440fe6d0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2441953a0 con 0x7fa2440fe6d0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22c03c8b0 con 0x7fa2440fe6d0 2024-09-13T15:26:40.377 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.374+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa244195840 con 0x7fa2440fe6d0 2024-09-13T15:26:40.379 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.375+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fa22c0035f0 con 0x7fa2440fe6d0 2024-09-13T15:26:40.379 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.375+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2440feb50 con 0x7fa2440fe6d0 2024-09-13T15:26:40.379 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.375+0000 7fa241ffb640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa230038740 0x7fa23003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:40.379 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.375+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa22c06c530 con 0x7fa2440fe6d0 2024-09-13T15:26:40.379 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.375+0000 7fa243fff640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa230038740 0x7fa23003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:40.379 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.376+0000 7fa243fff640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa230038740 0x7fa23003ac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa23400ad30 tx=0x7fa2340093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:40.381 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.380+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa22c04d350 con 0x7fa2440fe6d0 2024-09-13T15:26:40.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:40 smithi090 ceph-mon[27811]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:40.605 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.604+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa244104360 con 0x7fa2440fe6d0 2024-09-13T15:26:40.606 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.605+0000 7fa241ffb640 1 -- 172.21.15.175:0/3824509972 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa22c04a090 con 0x7fa2440fe6d0 2024-09-13T15:26:40.606 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:26:40.607 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:26:40.607 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:26:40.609 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa230038740 msgr2=0x7fa23003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:40.609 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa230038740 0x7fa23003ac00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa23400ad30 tx=0x7fa2340093f0 comp rx=0 tx=0).stop 2024-09-13T15:26:40.609 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 msgr2=0x7fa244194c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:40.609 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa244194c60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa22c009230 tx=0x7fa22c004f70 comp rx=0 tx=0).stop 2024-09-13T15:26:40.610 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 shutdown_connections 2024-09-13T15:26:40.610 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa230038740 0x7fa23003ac00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:40.610 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 --2- 172.21.15.175:0/3824509972 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa2440fe6d0 0x7fa244194c60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:40.610 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 >> 172.21.15.175:0/3824509972 conn(0x7fa2440f9e80 msgr2=0x7fa2440fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:40.610 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.607+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 shutdown_connections 2024-09-13T15:26:40.610 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:40.608+0000 7fa24ad39640 1 -- 172.21.15.175:0/3824509972 wait complete. 2024-09-13T15:26:41.487 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:41 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/3824509972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:26:42.195 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:42.196 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:42.391 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:42.411 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:42 smithi090 ceph-mon[27811]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:42.434 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:43.832 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 -- 172.21.15.175:0/4183249457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 msgr2=0x7fa6640fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 --2- 172.21.15.175:0/4183249457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa6640fead0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa654009c20 tx=0x7fa65402aa90 comp rx=0 tx=0).stop 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 -- 172.21.15.175:0/4183249457 shutdown_connections 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 --2- 172.21.15.175:0/4183249457 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa6640fead0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 -- 172.21.15.175:0/4183249457 >> 172.21.15.175:0/4183249457 conn(0x7fa6640f9e80 msgr2=0x7fa6640fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 -- 172.21.15.175:0/4183249457 shutdown_connections 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 -- 172.21.15.175:0/4183249457 wait complete. 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 Processor -- start 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.793+0000 7fa669769640 1 -- start start 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa669769640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa664194c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa669769640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6641951b0 con 0x7fa6640fe6d0 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa662ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa664194c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:43.833 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa662ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa664194c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:58924/0 (socket says 172.21.15.175:58924) 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa662ffd640 1 -- 172.21.15.175:0/1002926953 learned_addr learned my addr 172.21.15.175:0/1002926953 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa662ffd640 1 -- 172.21.15.175:0/1002926953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa654009900 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa662ffd640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa664194c70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa654006010 tx=0x7fa65402e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa65403e070 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.794+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6641953b0 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.795+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa65402fb00 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.795+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa654004630 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.795+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa664195850 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.795+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fa65400a040 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.795+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6640feb50 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.796+0000 7fa65bfff640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa650038740 0x7fa65003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.796+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa65406c5c0 con 0x7fa6640fe6d0 2024-09-13T15:26:43.834 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.796+0000 7fa6627fc640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa650038740 0x7fa65003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:43.835 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.796+0000 7fa6627fc640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa650038740 0x7fa65003ac00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0098c0 tx=0x7fa64c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:43.835 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:43.800+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa654004040 con 0x7fa6640fe6d0 2024-09-13T15:26:44.028 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.027+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa664104360 con 0x7fa6640fe6d0 2024-09-13T15:26:44.029 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.027+0000 7fa65bfff640 1 -- 172.21.15.175:0/1002926953 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fa654048090 con 0x7fa6640fe6d0 2024-09-13T15:26:44.029 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:26:44.029 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:26:44.030 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.029+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa650038740 msgr2=0x7fa65003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.029+0000 7fa669769640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa650038740 0x7fa65003ac00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa64c0098c0 tx=0x7fa64c006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.029+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 msgr2=0x7fa664194c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.029+0000 7fa669769640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa664194c70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa654006010 tx=0x7fa65402e940 comp rx=0 tx=0).stop 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.030+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 shutdown_connections 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.030+0000 7fa669769640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fa650038740 0x7fa65003ac00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.030+0000 7fa669769640 1 --2- 172.21.15.175:0/1002926953 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa6640fe6d0 0x7fa664194c70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.030+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 >> 172.21.15.175:0/1002926953 conn(0x7fa6640f9e80 msgr2=0x7fa664067f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:44.031 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.030+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 shutdown_connections 2024-09-13T15:26:44.032 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:44.030+0000 7fa669769640 1 -- 172.21.15.175:0/1002926953 wait complete. 2024-09-13T15:26:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:44 smithi090 ceph-mon[27811]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:44 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/1002926953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:26:45.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:45.549 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:45.742 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:45.786 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: Deploying daemon grafana.smithi090 on smithi090 2024-09-13T15:26:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:45 smithi090 ceph-mon[27811]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:47.233 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 -- 172.21.15.175:0/3873342525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 msgr2=0x7fdf840fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 --2- 172.21.15.175:0/3873342525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf840fead0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009880 tx=0x7fdf7802a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 -- 172.21.15.175:0/3873342525 shutdown_connections 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 --2- 172.21.15.175:0/3873342525 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf840fead0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 -- 172.21.15.175:0/3873342525 >> 172.21.15.175:0/3873342525 conn(0x7fdf840f9e80 msgr2=0x7fdf840fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 -- 172.21.15.175:0/3873342525 shutdown_connections 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.231+0000 7fdf8afda640 1 -- 172.21.15.175:0/3873342525 wait complete. 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.232+0000 7fdf8afda640 1 Processor -- start 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.232+0000 7fdf8afda640 1 -- start start 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.232+0000 7fdf8afda640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf84193b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:47.234 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.232+0000 7fdf8afda640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf841940b0 con 0x7fdf840fe6d0 2024-09-13T15:26:47.235 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.232+0000 7fdf88d76640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf84193b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:47.235 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf88d76640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf84193b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:58942/0 (socket says 172.21.15.175:58942) 2024-09-13T15:26:47.235 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf88d76640 1 -- 172.21.15.175:0/2663456783 learned_addr learned my addr 172.21.15.175:0/2663456783 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:47.236 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf88d76640 1 -- 172.21.15.175:0/2663456783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf78009560 con 0x7fdf840fe6d0 2024-09-13T15:26:47.236 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf88d76640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf84193b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009230 tx=0x7fdf780051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:47.236 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf7803e070 con 0x7fdf840fe6d0 2024-09-13T15:26:47.236 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdf78004c00 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf841942b0 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf780386b0 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.233+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf84194750 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.234+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fdf780388e0 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.234+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf840feb50 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.235+0000 7fdf81ffb640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdf74038740 0x7fdf7403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.235+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdf7806d530 con 0x7fdf840fe6d0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.235+0000 7fdf83fff640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdf74038740 0x7fdf7403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:47.237 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.235+0000 7fdf83fff640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdf74038740 0x7fdf7403ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf700098c0 tx=0x7fdf70006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:47.241 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.240+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fdf7804ab50 con 0x7fdf840fe6d0 2024-09-13T15:26:47.465 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.464+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf84104360 con 0x7fdf840fe6d0 2024-09-13T15:26:47.466 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.464+0000 7fdf81ffb640 1 -- 172.21.15.175:0/2663456783 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fdf78035090 con 0x7fdf840fe6d0 2024-09-13T15:26:47.466 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:26:47.466 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:26:47.466 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.466+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdf74038740 msgr2=0x7fdf7403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.466+0000 7fdf8afda640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdf74038740 0x7fdf7403ac00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf700098c0 tx=0x7fdf70006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 msgr2=0x7fdf84193b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf84193b70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009230 tx=0x7fdf780051a0 comp rx=0 tx=0).stop 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 shutdown_connections 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fdf74038740 0x7fdf7403ac00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 --2- 172.21.15.175:0/2663456783 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdf840fe6d0 0x7fdf84193b70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 >> 172.21.15.175:0/2663456783 conn(0x7fdf840f9e80 msgr2=0x7fdf840fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:47.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 shutdown_connections 2024-09-13T15:26:47.469 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:47.467+0000 7fdf8afda640 1 -- 172.21.15.175:0/2663456783 wait complete. 2024-09-13T15:26:48.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:48 smithi090 ceph-mon[27811]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:48.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:48 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:48.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:48 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2663456783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:26:49.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:49.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:49.301 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:49.345 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:50.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:50 smithi090 ceph-mon[27811]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:50.679 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 -- 172.21.15.175:0/1700736293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 msgr2=0x7f833c102710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 --2- 172.21.15.175:0/1700736293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c102710 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f832c009880 tx=0x7f832c02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 -- 172.21.15.175:0/1700736293 shutdown_connections 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 --2- 172.21.15.175:0/1700736293 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c102710 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 -- 172.21.15.175:0/1700736293 >> 172.21.15.175:0/1700736293 conn(0x7f833c0f9e80 msgr2=0x7f833c0fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 -- 172.21.15.175:0/1700736293 shutdown_connections 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.676+0000 7f83447ec640 1 -- 172.21.15.175:0/1700736293 wait complete. 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f83447ec640 1 Processor -- start 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f83447ec640 1 -- start start 2024-09-13T15:26:50.680 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f83447ec640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c194c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f83447ec640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833c195150 con 0x7f833c100320 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f8342588640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c194c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f8342588640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c194c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:57950/0 (socket says 172.21.15.175:57950) 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f8342588640 1 -- 172.21.15.175:0/230681623 learned_addr learned my addr 172.21.15.175:0/230681623 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f8342588640 1 -- 172.21.15.175:0/230681623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c009560 con 0x7f833c100320 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.677+0000 7f8342588640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c194c10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f832c009230 tx=0x7f832c004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.678+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c0055a0 con 0x7f833c100320 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.678+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c195350 con 0x7f833c100320 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.678+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f832c003a80 con 0x7f833c100320 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.678+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832c03c8b0 con 0x7f833c100320 2024-09-13T15:26:50.681 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.678+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c1957f0 con 0x7f833c100320 2024-09-13T15:26:50.682 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.679+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f832c0035f0 con 0x7f833c100320 2024-09-13T15:26:50.682 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.679+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c100780 con 0x7f833c100320 2024-09-13T15:26:50.682 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.679+0000 7f83337fe640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8328038740 0x7f832803ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:50.683 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.679+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f832c06c550 con 0x7f833c100320 2024-09-13T15:26:50.683 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.679+0000 7f8341d87640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8328038740 0x7f832803ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:50.683 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.680+0000 7f8341d87640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8328038740 0x7f832803ac00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83380098c0 tx=0x7f8338006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:50.685 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.684+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f832c03a9b0 con 0x7f833c100320 2024-09-13T15:26:50.907 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.905+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f833c18e380 con 0x7f833c100320 2024-09-13T15:26:50.907 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.906+0000 7f83337fe640 1 -- 172.21.15.175:0/230681623 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f833c18e380 con 0x7f833c100320 2024-09-13T15:26:50.907 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:26:50.908 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:26:50.908 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:26:50.909 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8328038740 msgr2=0x7f832803ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8328038740 0x7f832803ac00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83380098c0 tx=0x7f8338006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 msgr2=0x7f833c194c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c194c10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f832c009230 tx=0x7f832c004f70 comp rx=0 tx=0).stop 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 shutdown_connections 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8328038740 0x7f832803ac00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 --2- 172.21.15.175:0/230681623 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f833c100320 0x7f833c194c10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 >> 172.21.15.175:0/230681623 conn(0x7f833c0f9e80 msgr2=0x7f833c0faaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 shutdown_connections 2024-09-13T15:26:50.910 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:50.908+0000 7f83447ec640 1 -- 172.21.15.175:0/230681623 wait complete. 2024-09-13T15:26:51.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:51 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/230681623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:26:52.482 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:52.483 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:52.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:52 smithi090 ceph-mon[27811]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:52.680 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:52.723 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:54.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 -- 172.21.15.175:0/3001615941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 msgr2=0x7f29380feaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:54.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 --2- 172.21.15.175:0/3001615941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380feaf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2928009c20 tx=0x7f292802aa90 comp rx=0 tx=0).stop 2024-09-13T15:26:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 -- 172.21.15.175:0/3001615941 shutdown_connections 2024-09-13T15:26:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 --2- 172.21.15.175:0/3001615941 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380feaf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 -- 172.21.15.175:0/3001615941 >> 172.21.15.175:0/3001615941 conn(0x7f29380f9e80 msgr2=0x7f29380fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 -- 172.21.15.175:0/3001615941 shutdown_connections 2024-09-13T15:26:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 -- 172.21.15.175:0/3001615941 wait complete. 2024-09-13T15:26:54.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.200+0000 7f293dead640 1 Processor -- start 2024-09-13T15:26:54.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f293dead640 1 -- start start 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f293dead640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380fbd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f293dead640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29380fc280 con 0x7f29380fe6f0 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f29377fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380fbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f29377fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380fbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:57956/0 (socket says 172.21.15.175:57956) 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f29377fe640 1 -- 172.21.15.175:0/2014848365 learned_addr learned my addr 172.21.15.175:0/2014848365 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.201+0000 7f29377fe640 1 -- 172.21.15.175:0/2014848365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2928009900 con 0x7f29380fe6f0 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.202+0000 7f29377fe640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380fbd40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2928005cc0 tx=0x7f292802b020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.202+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f292803e070 con 0x7f29380fe6f0 2024-09-13T15:26:54.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.202+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29380fa470 con 0x7f29380fe6f0 2024-09-13T15:26:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.202+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f292802fb00 con 0x7f29380fe6f0 2024-09-13T15:26:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.202+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29280046a0 con 0x7f29380fe6f0 2024-09-13T15:26:54.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.202+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29380fa910 con 0x7f29380fe6f0 2024-09-13T15:26:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.203+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f292800a040 con 0x7f29380fe6f0 2024-09-13T15:26:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.203+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2938104c80 con 0x7f29380fe6f0 2024-09-13T15:26:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.203+0000 7f2934ff9640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f2920038740 0x7f292003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.203+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f292806c580 con 0x7f29380fe6f0 2024-09-13T15:26:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.204+0000 7f2936ffd640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f2920038740 0x7f292003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:54.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.204+0000 7f2936ffd640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f2920038740 0x7f292003ac00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f292c0098c0 tx=0x7f292c006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:54.209 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.208+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f2928004800 con 0x7f29380fe6f0 2024-09-13T15:26:54.437 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.435+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29380feb70 con 0x7f29380fe6f0 2024-09-13T15:26:54.437 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.436+0000 7f2934ff9640 1 -- 172.21.15.175:0/2014848365 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f2928048020 con 0x7f29380fe6f0 2024-09-13T15:26:54.437 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:26:54.438 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:26:54.438 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f2920038740 msgr2=0x7f292003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f2920038740 0x7f292003ac00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f292c0098c0 tx=0x7f292c006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 msgr2=0x7f29380fbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380fbd40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2928005cc0 tx=0x7f292802b020 comp rx=0 tx=0).stop 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 shutdown_connections 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f2920038740 0x7f292003ac00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 --2- 172.21.15.175:0/2014848365 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29380fe6f0 0x7f29380fbd40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 >> 172.21.15.175:0/2014848365 conn(0x7f29380f9e80 msgr2=0x7f29380686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:54.440 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 shutdown_connections 2024-09-13T15:26:54.441 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:54.438+0000 7f293dead640 1 -- 172.21.15.175:0/2014848365 wait complete. 2024-09-13T15:26:54.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:54 smithi090 ceph-mon[27811]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:55.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:55 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2014848365' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:26:56.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:56.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:56.130 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.131 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:56 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:56.259 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:56.302 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:57.434 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:57 smithi090 ceph-mon[27811]: Deploying daemon prometheus.smithi090 on smithi090 2024-09-13T15:26:57.434 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:57 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:26:57.730 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 -- 172.21.15.175:0/1891784475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 msgr2=0x7fb1641026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:57.730 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 --2- 172.21.15.175:0/1891784475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1641026f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb160009880 tx=0x7fb16002a5d0 comp rx=0 tx=0).stop 2024-09-13T15:26:57.730 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 -- 172.21.15.175:0/1891784475 shutdown_connections 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 --2- 172.21.15.175:0/1891784475 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1641026f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 -- 172.21.15.175:0/1891784475 >> 172.21.15.175:0/1891784475 conn(0x7fb1640f9e80 msgr2=0x7fb1640fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 -- 172.21.15.175:0/1891784475 shutdown_connections 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.727+0000 7fb16c259640 1 -- 172.21.15.175:0/1891784475 wait complete. 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb16c259640 1 Processor -- start 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb16c259640 1 -- start start 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb16c259640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1640ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb16c259640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1641000c0 con 0x7fb164100300 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb169ff5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1640ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb169ff5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1640ffb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:57978/0 (socket says 172.21.15.175:57978) 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb169ff5640 1 -- 172.21.15.175:0/381097326 learned_addr learned my addr 172.21.15.175:0/381097326 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:26:57.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb169ff5640 1 -- 172.21.15.175:0/381097326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160009560 con 0x7fb164100300 2024-09-13T15:26:57.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.728+0000 7fb169ff5640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1640ffb80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb160009230 tx=0x7fb160004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:57.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.729+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1600055a0 con 0x7fb164100300 2024-09-13T15:26:57.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.729+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb160003a80 con 0x7fb164100300 2024-09-13T15:26:57.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.729+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1640fe2b0 con 0x7fb164100300 2024-09-13T15:26:57.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.729+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb16003c8b0 con 0x7fb164100300 2024-09-13T15:26:57.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.729+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1640fe750 con 0x7fb164100300 2024-09-13T15:26:57.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.730+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fb1600035f0 con 0x7fb164100300 2024-09-13T15:26:57.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.730+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb164105d40 con 0x7fb164100300 2024-09-13T15:26:57.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.730+0000 7fb15affd640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb150038740 0x7fb15003ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:26:57.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.730+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb16006c4b0 con 0x7fb164100300 2024-09-13T15:26:57.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.730+0000 7fb1697f4640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb150038740 0x7fb15003ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:26:57.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.730+0000 7fb1697f4640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb150038740 0x7fb15003ac00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb1540098c0 tx=0x7fb154006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:26:57.736 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.735+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb16004d350 con 0x7fb164100300 2024-09-13T15:26:57.958 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.957+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb164100760 con 0x7fb164100300 2024-09-13T15:26:57.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.958+0000 7fb15affd640 1 -- 172.21.15.175:0/381097326 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fb16004a090 con 0x7fb164100300 2024-09-13T15:26:57.959 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:26:57.960 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:26:57.960 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:26:57.961 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb150038740 msgr2=0x7fb15003ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:57.961 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb150038740 0x7fb15003ac00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb1540098c0 tx=0x7fb154006db0 comp rx=0 tx=0).stop 2024-09-13T15:26:57.961 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 msgr2=0x7fb1640ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1640ffb80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb160009230 tx=0x7fb160004f70 comp rx=0 tx=0).stop 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 shutdown_connections 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fb150038740 0x7fb15003ac00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 --2- 172.21.15.175:0/381097326 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb164100300 0x7fb1640ffb80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 >> 172.21.15.175:0/381097326 conn(0x7fb1640f9e80 msgr2=0x7fb1640fab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 shutdown_connections 2024-09-13T15:26:57.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:26:57.960+0000 7fb16c259640 1 -- 172.21.15.175:0/381097326 wait complete. 2024-09-13T15:26:58.410 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:58 smithi090 ceph-mon[27811]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:26:58.410 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:26:58 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/381097326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:26:59.507 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:26:59.507 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:26:59.690 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:26:59.734 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:00.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:00 smithi090 ceph-mon[27811]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:01.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3302930165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 msgr2=0x7f8aa40feaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 --2- 172.21.15.175:0/3302930165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa40feaf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009880 tx=0x7f8aa002a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3302930165 shutdown_connections 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 --2- 172.21.15.175:0/3302930165 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa40feaf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3302930165 >> 172.21.15.175:0/3302930165 conn(0x7f8aa40f9e80 msgr2=0x7f8aa40fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3302930165 shutdown_connections 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3302930165 wait complete. 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 Processor -- start 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.197+0000 7f8aaad0f640 1 -- start start 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aaad0f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa4194bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aaad0f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa4195100 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aa8aab640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa4194bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:01.203 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aa8aab640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa4194bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:46766/0 (socket says 172.21.15.175:46766) 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aa8aab640 1 -- 172.21.15.175:0/3199961038 learned_addr learned my addr 172.21.15.175:0/3199961038 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aa8aab640 1 -- 172.21.15.175:0/3199961038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0009560 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.198+0000 7f8aa8aab640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa4194bc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009230 tx=0x7f8aa00051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.199+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa00035f0 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.199+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa4195300 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.199+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8aa0003be0 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.199+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa003c980 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.199+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa41957a0 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.200+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa40feb70 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.200+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f8aa0003750 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.200+0000 7f8a99ffb640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8a8c038740 0x7f8a8c03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.200+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8aa006c430 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.200+0000 7f8a9bfff640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8a8c038740 0x7f8a8c03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:01.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.201+0000 7f8a9bfff640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8a8c038740 0x7f8a8c03ac00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8a940098c0 tx=0x7f8a94006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:01.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.205+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f8aa003a540 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.429 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.428+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8aa418e2d0 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.429 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.428+0000 7f8a99ffb640 1 -- 172.21.15.175:0/3199961038 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f8aa0035090 con 0x7f8aa40fe6f0 2024-09-13T15:27:01.430 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:01.430 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:01.430 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8a8c038740 msgr2=0x7f8a8c03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8a8c038740 0x7f8a8c03ac00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8a940098c0 tx=0x7f8a94006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 msgr2=0x7f8aa4194bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa4194bc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8aa0009230 tx=0x7f8aa00051a0 comp rx=0 tx=0).stop 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 shutdown_connections 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f8a8c038740 0x7f8a8c03ac00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 --2- 172.21.15.175:0/3199961038 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8aa40fe6f0 0x7f8aa4194bc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 >> 172.21.15.175:0/3199961038 conn(0x7f8aa40f9e80 msgr2=0x7f8aa40686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 shutdown_connections 2024-09-13T15:27:01.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:01.431+0000 7f8aaad0f640 1 -- 172.21.15.175:0/3199961038 wait complete. 2024-09-13T15:27:02.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:02 smithi090 ceph-mon[27811]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:02.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:02 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/3199961038' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:02.979 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:02.979 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:03.185 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:03.228 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:04.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:04 smithi090 ceph-mon[27811]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:04.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 -- 172.21.15.175:0/1871494892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 msgr2=0x7fc568072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:04.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 --2- 172.21.15.175:0/1871494892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc568072940 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc560009880 tx=0x7fc56002a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:04.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 -- 172.21.15.175:0/1871494892 shutdown_connections 2024-09-13T15:27:04.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 --2- 172.21.15.175:0/1871494892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc568072940 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:04.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 -- 172.21.15.175:0/1871494892 >> 172.21.15.175:0/1871494892 conn(0x7fc5680faf90 msgr2=0x7fc5680fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:04.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 -- 172.21.15.175:0/1871494892 shutdown_connections 2024-09-13T15:27:04.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 -- 172.21.15.175:0/1871494892 wait complete. 2024-09-13T15:27:04.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 Processor -- start 2024-09-13T15:27:04.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.797+0000 7fc56f4f1640 1 -- start start 2024-09-13T15:27:04.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56f4f1640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc56819b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:04.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56f4f1640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56819b6c0 con 0x7fc5680744e0 2024-09-13T15:27:04.804 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56d28d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc56819b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:04.804 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56d28d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc56819b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:46784/0 (socket says 172.21.15.175:46784) 2024-09-13T15:27:04.804 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56d28d640 1 -- 172.21.15.175:0/4258641906 learned_addr learned my addr 172.21.15.175:0/4258641906 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:04.804 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56d28d640 1 -- 172.21.15.175:0/4258641906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc560009560 con 0x7fc5680744e0 2024-09-13T15:27:04.804 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc56d28d640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc56819b180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc560009230 tx=0x7fc5600051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:04.805 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.798+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc5600035f0 con 0x7fc5680744e0 2024-09-13T15:27:04.805 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.799+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56819b8c0 con 0x7fc5680744e0 2024-09-13T15:27:04.805 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.799+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc560003be0 con 0x7fc5680744e0 2024-09-13T15:27:04.806 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.799+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc56003c980 con 0x7fc5680744e0 2024-09-13T15:27:04.806 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.799+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56819bd60 con 0x7fc5680744e0 2024-09-13T15:27:04.806 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.799+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7fc560003750 con 0x7fc5680744e0 2024-09-13T15:27:04.806 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.799+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5680729c0 con 0x7fc5680744e0 2024-09-13T15:27:04.806 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.800+0000 7fc55a7fc640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc554038740 0x7fc55403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:04.806 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.800+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc56006c4b0 con 0x7fc5680744e0 2024-09-13T15:27:04.807 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.800+0000 7fc56ca8c640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc554038740 0x7fc55403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:04.807 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.800+0000 7fc56ca8c640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc554038740 0x7fc55403ac00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc56400ad30 tx=0x7fc5640093f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:04.808 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:04.804+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fc56003a540 con 0x7fc5680744e0 2024-09-13T15:27:05.029 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.028+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc5681949a0 con 0x7fc5680744e0 2024-09-13T15:27:05.030 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.029+0000 7fc55a7fc640 1 -- 172.21.15.175:0/4258641906 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fc560035090 con 0x7fc5680744e0 2024-09-13T15:27:05.030 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:05.031 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:05.031 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:05.032 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc554038740 msgr2=0x7fc55403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:05.032 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc554038740 0x7fc55403ac00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc56400ad30 tx=0x7fc5640093f0 comp rx=0 tx=0).stop 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 msgr2=0x7fc56819b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc56819b180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc560009230 tx=0x7fc5600051a0 comp rx=0 tx=0).stop 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 shutdown_connections 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7fc554038740 0x7fc55403ac00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 --2- 172.21.15.175:0/4258641906 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc5680744e0 0x7fc56819b180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 >> 172.21.15.175:0/4258641906 conn(0x7fc5680faf90 msgr2=0x7fc5680fba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 shutdown_connections 2024-09-13T15:27:05.033 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:05.031+0000 7fc56f4f1640 1 -- 172.21.15.175:0/4258641906 wait complete. 2024-09-13T15:27:05.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:05 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/4258641906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:06.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:06 smithi090 ceph-mon[27811]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:06.607 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:06.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:06.802 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:06.845 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:08 smithi090 ceph-mon[27811]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:08.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 -- 172.21.15.175:0/3133363441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 msgr2=0x7f29f00fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:08.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 --2- 172.21.15.175:0/3133363441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f00fead0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009880 tx=0x7f29ec02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:08.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 -- 172.21.15.175:0/3133363441 shutdown_connections 2024-09-13T15:27:08.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 --2- 172.21.15.175:0/3133363441 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f00fead0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:08.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 -- 172.21.15.175:0/3133363441 >> 172.21.15.175:0/3133363441 conn(0x7f29f00f9e80 msgr2=0x7f29f00fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 -- 172.21.15.175:0/3133363441 shutdown_connections 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 -- 172.21.15.175:0/3133363441 wait complete. 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 Processor -- start 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.264+0000 7f29f7760640 1 -- start start 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f7760640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f0194c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f7760640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f01951a0 con 0x7f29f00fe6d0 2024-09-13T15:27:08.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f54fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f0194c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:08.269 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f54fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f0194c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:46792/0 (socket says 172.21.15.175:46792) 2024-09-13T15:27:08.269 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f54fc640 1 -- 172.21.15.175:0/2317745700 learned_addr learned my addr 172.21.15.175:0/2317745700 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:08.269 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f54fc640 1 -- 172.21.15.175:0/2317745700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29ec009560 con 0x7f29f00fe6d0 2024-09-13T15:27:08.269 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29f54fc640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f0194c60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009230 tx=0x7f29ec004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.265+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ec0055a0 con 0x7f29f00fe6d0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.266+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f01953a0 con 0x7f29f00fe6d0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.266+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f29ec003a80 con 0x7f29f00fe6d0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.266+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29ec03c8b0 con 0x7f29f00fe6d0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.266+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f0195840 con 0x7f29f00fe6d0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.266+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f29ec03cb30 con 0x7f29f00fe6d0 2024-09-13T15:27:08.270 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.266+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f00feb50 con 0x7f29f00fe6d0 2024-09-13T15:27:08.271 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.267+0000 7f29e67fc640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f29dc038740 0x7f29dc03ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:08.271 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.267+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f29ec06c510 con 0x7f29f00fe6d0 2024-09-13T15:27:08.271 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.267+0000 7f29f4cfb640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f29dc038740 0x7f29dc03ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:08.271 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.267+0000 7f29f4cfb640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f29dc038740 0x7f29dc03ac00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f29e00098c0 tx=0x7f29e0006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:08.272 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.271+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f29ec03bda0 con 0x7f29f00fe6d0 2024-09-13T15:27:08.501 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.500+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29f0104360 con 0x7f29f00fe6d0 2024-09-13T15:27:08.501 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.500+0000 7f29e67fc640 1 -- 172.21.15.175:0/2317745700 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f29ec040090 con 0x7f29f00fe6d0 2024-09-13T15:27:08.502 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:08.502 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:08.502 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:08.504 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f29dc038740 msgr2=0x7f29dc03ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f29dc038740 0x7f29dc03ac00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f29e00098c0 tx=0x7f29e0006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 msgr2=0x7f29f0194c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f0194c60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009230 tx=0x7f29ec004f70 comp rx=0 tx=0).stop 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 shutdown_connections 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f29dc038740 0x7f29dc03ac00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 --2- 172.21.15.175:0/2317745700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f29f00fe6d0 0x7f29f0194c60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 >> 172.21.15.175:0/2317745700 conn(0x7f29f00f9e80 msgr2=0x7f29f00fabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 shutdown_connections 2024-09-13T15:27:08.505 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:08.503+0000 7f29f7760640 1 -- 172.21.15.175:0/2317745700 wait complete. 2024-09-13T15:27:09.426 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:09 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2317745700' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:10.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:10.037 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:10.231 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:10.275 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:10.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:10 smithi090 ceph-mon[27811]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.700+0000 7f97ff426640 1 -- 172.21.15.175:0/2832987014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 msgr2=0x7f97f80ffbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.700+0000 7f97ff426640 1 --2- 172.21.15.175:0/2832987014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f80ffbe0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009880 tx=0x7f97f402a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 -- 172.21.15.175:0/2832987014 shutdown_connections 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 --2- 172.21.15.175:0/2832987014 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f80ffbe0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 -- 172.21.15.175:0/2832987014 >> 172.21.15.175:0/2832987014 conn(0x7f97f80faf90 msgr2=0x7f97f80fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 -- 172.21.15.175:0/2832987014 shutdown_connections 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 -- 172.21.15.175:0/2832987014 wait complete. 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 Processor -- start 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.701+0000 7f97ff426640 1 -- start start 2024-09-13T15:27:11.706 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97ff426640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f8194c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97ff426640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f8195160 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97fd1c2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f8194c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97fd1c2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f8194c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:36264/0 (socket says 172.21.15.175:36264) 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97fd1c2640 1 -- 172.21.15.175:0/378293354 learned_addr learned my addr 172.21.15.175:0/378293354 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97fd1c2640 1 -- 172.21.15.175:0/378293354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f4009560 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97fd1c2640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f8194c20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009230 tx=0x7f97f40051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97f40035f0 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f97f4003be0 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.702+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f8195360 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.703+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f97f403c980 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.703+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f8195800 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.703+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f80ffc60 con 0x7f97f80ff7e0 2024-09-13T15:27:11.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.704+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 13) v1 ==== 45612+0+0 (secure 0 0 0) 0x7f97f4003750 con 0x7f97f80ff7e0 2024-09-13T15:27:11.708 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.704+0000 7f97ee7fc640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f97e4038740 0x7f97e403ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:11.708 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.704+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f97f406c430 con 0x7f97f80ff7e0 2024-09-13T15:27:11.708 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.704+0000 7f97fc9c1640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f97e4038740 0x7f97e403ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:11.708 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.705+0000 7f97fc9c1640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f97e4038740 0x7f97e403ac00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97e80098c0 tx=0x7f97e8006db0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:11.710 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.709+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f97f403a540 con 0x7f97f80ff7e0 2024-09-13T15:27:11.931 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.930+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f97f818e3b0 con 0x7f97f80ff7e0 2024-09-13T15:27:11.932 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.931+0000 7f97ee7fc640 1 -- 172.21.15.175:0/378293354 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f97f4035090 con 0x7f97f80ff7e0 2024-09-13T15:27:11.932 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:11.932 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:11.933 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:11.934 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f97e4038740 msgr2=0x7f97e403ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:11.934 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f97e4038740 0x7f97e403ac00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97e80098c0 tx=0x7f97e8006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:11.934 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 msgr2=0x7f97f8194c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f8194c20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f97f4009230 tx=0x7f97f40051a0 comp rx=0 tx=0).stop 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 shutdown_connections 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f97e4038740 0x7f97e403ac00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 --2- 172.21.15.175:0/378293354 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f97f80ff7e0 0x7f97f8194c20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 >> 172.21.15.175:0/378293354 conn(0x7f97f80faf90 msgr2=0x7f97f80fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 shutdown_connections 2024-09-13T15:27:11.935 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:11.933+0000 7f97ff426640 1 -- 172.21.15.175:0/378293354 wait complete. 2024-09-13T15:27:12.431 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:12 smithi090 ceph-mon[27811]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:12.431 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:12 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/378293354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:13.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:13.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:13.711 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:13.754 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:14.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:13 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:14.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:13 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:14.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:13 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:14.161 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:13 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-13T15:27:14.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:13 smithi090 ceph-mon[27811]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:15.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:14 smithi090 ceph-mon[27811]: from='mgr.14162 172.21.15.90:0/1684508625' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-13T15:27:15.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:14 smithi090 ceph-mon[27811]: mgrmap e14: smithi090.fyvvpd(active, since 66s) 2024-09-13T15:27:15.307 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2826775097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 msgr2=0x7f6cec0fdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:15.307 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 --2- 172.21.15.175:0/2826775097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec0fdea0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009880 tx=0x7f6cdc02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:15.307 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2826775097 shutdown_connections 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 --2- 172.21.15.175:0/2826775097 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec0fdea0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2826775097 >> 172.21.15.175:0/2826775097 conn(0x7f6cec0f9610 msgr2=0x7f6cec0fba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2826775097 shutdown_connections 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.304+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2826775097 wait complete. 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf43ba640 1 Processor -- start 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf43ba640 1 -- start start 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf43ba640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec194bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf43ba640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec195130 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf2156640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec194bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf2156640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec194bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:36272/0 (socket says 172.21.15.175:36272) 2024-09-13T15:27:15.308 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.305+0000 7f6cf2156640 1 -- 172.21.15.175:0/2927249554 learned_addr learned my addr 172.21.15.175:0/2927249554 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6cf2156640 1 -- 172.21.15.175:0/2927249554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc009560 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6cf2156640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec194bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009230 tx=0x7f6cdc0051a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cdc0035f0 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6cdc003be0 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec195330 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cdc03c980 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.309 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.306+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec1957d0 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.310 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.307+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 14) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f6cdc003750 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.310 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.307+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec0fdf20 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.310 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.307+0000 7f6ce37fe640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 0x7f6cd803ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:15.310 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.307+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6cdc06c540 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.310 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.307+0000 7f6cf1955640 1 -- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 msgr2=0x7f6cd803ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3319538530 2024-09-13T15:27:15.310 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.307+0000 7f6cf1955640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 0x7f6cd803ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T15:27:15.313 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.312+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6cdc03a9b0 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.508+0000 7f6cf1955640 1 -- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 msgr2=0x7f6cd803ac50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.90:6800/3319538530 2024-09-13T15:27:15.509 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.508+0000 7f6cf1955640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 0x7f6cd803ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T15:27:15.539 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.538+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cec18e3a0 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.539 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.538+0000 7f6ce37fe640 1 -- 172.21.15.175:0/2927249554 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f6cdc035070 con 0x7f6cec0fdaa0 2024-09-13T15:27:15.540 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:15.540 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:15.540 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:15.546 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 msgr2=0x7f6cd803ac50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T15:27:15.546 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 0x7f6cd803ac50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:15.546 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 msgr2=0x7f6cec194bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:15.546 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec194bf0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009230 tx=0x7f6cdc0051a0 comp rx=0 tx=0).stop 2024-09-13T15:27:15.547 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 shutdown_connections 2024-09-13T15:27:15.547 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:6800/3319538530,v1:172.21.15.90:6801/3319538530] conn(0x7f6cd8038790 0x7f6cd803ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:15.547 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 --2- 172.21.15.175:0/2927249554 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6cec0fdaa0 0x7f6cec194bf0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:15.547 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 >> 172.21.15.175:0/2927249554 conn(0x7f6cec0f9610 msgr2=0x7f6cec0fa1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:15.547 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 shutdown_connections 2024-09-13T15:27:15.547 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:15.540+0000 7f6cf43ba640 1 -- 172.21.15.175:0/2927249554 wait complete. 2024-09-13T15:27:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2927249554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:17.119 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:17.120 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:17.322 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:17.366 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:18.746 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 -- 172.21.15.175:0/2372615552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 msgr2=0x7f94500fead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 --2- 172.21.15.175:0/2372615552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f94500fead0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9444009880 tx=0x7f944402a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 -- 172.21.15.175:0/2372615552 shutdown_connections 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 --2- 172.21.15.175:0/2372615552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f94500fead0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 -- 172.21.15.175:0/2372615552 >> 172.21.15.175:0/2372615552 conn(0x7f94500f9e80 msgr2=0x7f94500fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 -- 172.21.15.175:0/2372615552 shutdown_connections 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.744+0000 7f9456d3d640 1 -- 172.21.15.175:0/2372615552 wait complete. 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.745+0000 7f9456d3d640 1 Processor -- start 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.745+0000 7f9456d3d640 1 -- start start 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.745+0000 7f9456d3d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f9450193b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:18.747 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.745+0000 7f9456d3d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94501940b0 con 0x7f94500fe6d0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f9454ad9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f9450193b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f9454ad9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f9450193b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:38524/0 (socket says 172.21.15.175:38524) 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f9454ad9640 1 -- 172.21.15.175:0/697222140 learned_addr learned my addr 172.21.15.175:0/697222140 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f9454ad9640 1 -- 172.21.15.175:0/697222140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9444009560 con 0x7f94500fe6d0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f9454ad9640 1 --2- 172.21.15.175:0/697222140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f9450193b70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9444009230 tx=0x7f9444004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94440055a0 con 0x7f94500fe6d0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.746+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9444003a80 con 0x7f94500fe6d0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.747+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94501942b0 con 0x7f94500fe6d0 2024-09-13T15:27:18.749 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.747+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f944403c8b0 con 0x7f94500fe6d0 2024-09-13T15:27:18.750 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.747+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9450194690 con 0x7f94500fe6d0 2024-09-13T15:27:18.750 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.748+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 15) v1 ==== 45350+0+0 (secure 0 0 0) 0x7f94440035f0 con 0x7f94500fe6d0 2024-09-13T15:27:18.750 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.748+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f944406c190 con 0x7f94500fe6d0 2024-09-13T15:27:18.750 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.748+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9450104c60 con 0x7f94500fe6d0 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: Active manager daemon smithi090.fyvvpd restarted 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: Activating manager daemon smithi090.fyvvpd 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: osdmap e5: 0 total, 0 up, 0 in 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: mgrmap e15: smithi090.fyvvpd(active, starting, since 0.00296301s) 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr metadata", "who": "smithi090.fyvvpd", "id": "smithi090.fyvvpd"}]: dispatch 2024-09-13T15:27:18.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: Manager daemon smithi090.fyvvpd is now available 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:27:18.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/mirror_snapshot_schedule"}]: dispatch 2024-09-13T15:27:18.754 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.753+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f944404a080 con 0x7f94500fe6d0 2024-09-13T15:27:18.986 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.984+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f945018cdb0 con 0x7f94500fe6d0 2024-09-13T15:27:18.986 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.985+0000 7f944dffb640 1 -- 172.21.15.175:0/697222140 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f9444035020 con 0x7f94500fe6d0 2024-09-13T15:27:18.987 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:18.987 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:18.987 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:18.988 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 msgr2=0x7f9450193b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:18.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 --2- 172.21.15.175:0/697222140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f9450193b70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9444009230 tx=0x7f9444004f70 comp rx=0 tx=0).stop 2024-09-13T15:27:18.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 shutdown_connections 2024-09-13T15:27:18.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 --2- 172.21.15.175:0/697222140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f94500fe6d0 0x7f9450193b70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:18.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 >> 172.21.15.175:0/697222140 conn(0x7f94500f9e80 msgr2=0x7f94500fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:18.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 shutdown_connections 2024-09-13T15:27:18.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:18.987+0000 7f9456d3d640 1 -- 172.21.15.175:0/697222140 wait complete. 2024-09-13T15:27:19.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi090.fyvvpd/trash_purge_schedule"}]: dispatch 2024-09-13T15:27:19.799 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:19 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/697222140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:19.799 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:19.799 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:19 smithi090 ceph-mon[27811]: mgrmap e16: smithi090.fyvvpd(active, since 1.0059s) 2024-09-13T15:27:20.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:20.627 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:20.969 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:20 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:20.969 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:20 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:21.477 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:21.878 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-13T15:27:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:21 smithi090 ceph-mon[27811]: [13/Sep/2024:15:27:20] ENGINE Bus STARTING 2024-09-13T15:27:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:21 smithi090 ceph-mon[27811]: [13/Sep/2024:15:27:20] ENGINE Serving on https://172.21.15.90:7150 2024-09-13T15:27:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:21 smithi090 ceph-mon[27811]: [13/Sep/2024:15:27:20] ENGINE Error in HTTPServer.serve 2024-09-13T15:27:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: Traceback (most recent call last): 2024-09-13T15:27:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-13T15:27:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._connections.run(self.expiration_interval) 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._run(expiration_interval) 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: s = self.context.wrap_socket( 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: return self.sslsocket_class._create( 2024-09-13T15:27:22.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self.do_handshake() 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: self._sslobj.do_handshake() 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:21 smithi090 ceph-mon[27811]: [13/Sep/2024:15:27:20] ENGINE Serving on http://172.21.15.90:8765 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:21 smithi090 ceph-mon[27811]: [13/Sep/2024:15:27:20] ENGINE Bus STARTED 2024-09-13T15:27:22.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:21 smithi090 ceph-mon[27811]: mgrmap e17: smithi090.fyvvpd(active, since 2s) 2024-09-13T15:27:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd/host:smithi090", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd/host:smithi175", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:24.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:27:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.189+0000 7f3401c9b640 1 -- 172.21.15.175:0/2329550978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 msgr2=0x7f33fc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 --2- 172.21.15.175:0/2329550978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc104a30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009880 tx=0x7f33ec02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 -- 172.21.15.175:0/2329550978 shutdown_connections 2024-09-13T15:27:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 --2- 172.21.15.175:0/2329550978 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc104a30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:24.194 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 -- 172.21.15.175:0/2329550978 >> 172.21.15.175:0/2329550978 conn(0x7f33fc075490 msgr2=0x7f33fc0fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 -- 172.21.15.175:0/2329550978 shutdown_connections 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 -- 172.21.15.175:0/2329550978 wait complete. 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.190+0000 7f3401c9b640 1 Processor -- start 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f3401c9b640 1 -- start start 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f3401c9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc193eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f3401c9b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc1943f0 con 0x7f33fc104650 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc193eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f33fb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc193eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:38546/0 (socket says 172.21.15.175:38546) 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f33fb7fe640 1 -- 172.21.15.175:0/3223664320 learned_addr learned my addr 172.21.15.175:0/3223664320 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.191+0000 7f33fb7fe640 1 -- 172.21.15.175:0/3223664320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec009560 con 0x7f33fc104650 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.192+0000 7f33fb7fe640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc193eb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009230 tx=0x7f33ec004e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:24.195 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.192+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec0054c0 con 0x7f33fc104650 2024-09-13T15:27:24.196 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.192+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f33ec005620 con 0x7f33fc104650 2024-09-13T15:27:24.196 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.192+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc194650 con 0x7f33fc104650 2024-09-13T15:27:24.196 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.192+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33ec03c890 con 0x7f33fc104650 2024-09-13T15:27:24.196 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.192+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc194ad0 con 0x7f33fc104650 2024-09-13T15:27:24.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.193+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f33ec03c9f0 con 0x7f33fc104650 2024-09-13T15:27:24.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.193+0000 7f33f8ff9640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f33e4038880 0x7f33e403ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:24.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.193+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f33ec06c5e0 con 0x7f33fc104650 2024-09-13T15:27:24.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.193+0000 7f33faffd640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f33e4038880 0x7f33e403ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:24.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.193+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33fc0ffd90 con 0x7f33fc104650 2024-09-13T15:27:24.197 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.194+0000 7f33faffd640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f33e4038880 0x7f33e403ad40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f33f00098c0 tx=0x7f33f0006db0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:24.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.198+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f33ec035080 con 0x7f33fc104650 2024-09-13T15:27:24.447 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.445+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33fc108870 con 0x7f33fc104650 2024-09-13T15:27:24.448 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.445+0000 7f33f8ff9640 1 -- 172.21.15.175:0/3223664320 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f33ec040020 con 0x7f33fc104650 2024-09-13T15:27:24.448 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:24.448 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:24.448 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:24.450 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f33e4038880 msgr2=0x7f33e403ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f33e4038880 0x7f33e403ad40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f33f00098c0 tx=0x7f33f0006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 msgr2=0x7f33fc193eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc193eb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f33ec009230 tx=0x7f33ec004e90 comp rx=0 tx=0).stop 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 shutdown_connections 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f33e4038880 0x7f33e403ad40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 --2- 172.21.15.175:0/3223664320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f33fc104650 0x7f33fc193eb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 >> 172.21.15.175:0/3223664320 conn(0x7f33fc075490 msgr2=0x7f33fc0fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 shutdown_connections 2024-09-13T15:27:24.451 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:24.448+0000 7f3401c9b640 1 -- 172.21.15.175:0/3223664320 wait complete. 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi090:/etc/ceph/ceph.conf 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi090:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi175:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi090:/etc/ceph/ceph.client.admin.keyring 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi175:/etc/ceph/ceph.client.admin.keyring 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi090:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: Updating smithi175:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi175", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi175", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]': finished 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:24.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:24 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/3223664320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:25 smithi090 ceph-mon[27811]: Deploying daemon ceph-exporter.smithi175 on smithi175 2024-09-13T15:27:26.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:26.022 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:26.335 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.187+0000 7fcfffa1f640 1 -- 172.21.15.175:0/1865506232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 msgr2=0x7fcff8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.187+0000 7fcfffa1f640 1 --2- 172.21.15.175:0/1865506232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8104a30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009880 tx=0x7fcff402a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 -- 172.21.15.175:0/1865506232 shutdown_connections 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 --2- 172.21.15.175:0/1865506232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8104a30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 -- 172.21.15.175:0/1865506232 >> 172.21.15.175:0/1865506232 conn(0x7fcff8075490 msgr2=0x7fcff80fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 -- 172.21.15.175:0/1865506232 shutdown_connections 2024-09-13T15:27:28.216 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 -- 172.21.15.175:0/1865506232 wait complete. 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 Processor -- start 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 -- start start 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8193c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.188+0000 7fcfffa1f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff80684e0 con 0x7fcff8104650 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcffd7bb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8193c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcffd7bb640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8193c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:38564/0 (socket says 172.21.15.175:38564) 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcffd7bb640 1 -- 172.21.15.175:0/4147111298 learned_addr learned my addr 172.21.15.175:0/4147111298 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcffd7bb640 1 -- 172.21.15.175:0/4147111298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff4009560 con 0x7fcff8104650 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcffd7bb640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8193c10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009230 tx=0x7fcff4004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff40053e0 con 0x7fcff8104650 2024-09-13T15:27:28.217 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcff4005540 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcff403c7e0 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff8194150 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.189+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff81945d0 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.190+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7fcff40035f0 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.190+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff80ffd90 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.190+0000 7fcfee7fc640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcfe4038830 0x7fcfe403acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.190+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7fcff406c6b0 con 0x7fcff8104650 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.191+0000 7fcffcfba640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcfe4038830 0x7fcfe403acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.191+0000 7fcffcfba640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcfe4038830 0x7fcfe403acf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80098c0 tx=0x7fcfe8006db0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:28.218 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.195+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fcff403a540 con 0x7fcff8104650 2024-09-13T15:27:28.430 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.427+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcff8067db0 con 0x7fcff8104650 2024-09-13T15:27:28.430 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.428+0000 7fcfee7fc640 1 -- 172.21.15.175:0/4147111298 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7fcff404a300 con 0x7fcff8104650 2024-09-13T15:27:28.431 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:28.431 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:28.431 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcfe4038830 msgr2=0x7fcfe403acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcfe4038830 0x7fcfe403acf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80098c0 tx=0x7fcfe8006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 msgr2=0x7fcff8193c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:28.432 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8193c10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcff4009230 tx=0x7fcff4004e70 comp rx=0 tx=0).stop 2024-09-13T15:27:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 shutdown_connections 2024-09-13T15:27:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcfe4038830 0x7fcfe403acf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 --2- 172.21.15.175:0/4147111298 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcff8104650 0x7fcff8193c10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 >> 172.21.15.175:0/4147111298 conn(0x7fcff8075490 msgr2=0x7fcff80fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 shutdown_connections 2024-09-13T15:27:28.433 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:28.430+0000 7fcfffa1f640 1 -- 172.21.15.175:0/4147111298 wait complete. 2024-09-13T15:27:28.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:28 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/4147111298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:29.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:29.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:29.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:29.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:29.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi175", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-13T15:27:29.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi175", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-13T15:27:29.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:29.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: Deploying daemon crash.smithi175 on smithi175 2024-09-13T15:27:29.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:29.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:29.999 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:30.199 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 -- 172.21.15.175:0/4209954369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 msgr2=0x7f0780100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 --2- 172.21.15.175:0/4209954369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f0780100d70 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0770009880 tx=0x7f077002a670 comp rx=0 tx=0).stop 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 -- 172.21.15.175:0/4209954369 shutdown_connections 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 --2- 172.21.15.175:0/4209954369 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f0780100d70 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 -- 172.21.15.175:0/4209954369 >> 172.21.15.175:0/4209954369 conn(0x7f07800fa3b0 msgr2=0x7f07800fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 -- 172.21.15.175:0/4209954369 shutdown_connections 2024-09-13T15:27:32.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.237+0000 7f0785e52640 1 -- 172.21.15.175:0/4209954369 wait complete. 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.238+0000 7f0785e52640 1 Processor -- start 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.238+0000 7f0785e52640 1 -- start start 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.238+0000 7f0785e52640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f078019b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.238+0000 7f0785e52640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078019bab0 con 0x7f0780100990 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.239+0000 7f077f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f078019b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.239+0000 7f077f7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f078019b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:60666/0 (socket says 172.21.15.175:60666) 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.239+0000 7f077f7fe640 1 -- 172.21.15.175:0/221135550 learned_addr learned my addr 172.21.15.175:0/221135550 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.239+0000 7f077f7fe640 1 -- 172.21.15.175:0/221135550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0770009560 con 0x7f0780100990 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.239+0000 7f077f7fe640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f078019b570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0770009230 tx=0x7f0770004e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.239+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07700054c0 con 0x7f0780100990 2024-09-13T15:27:32.265 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.240+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0770005620 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.240+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078019bcb0 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.240+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f077003c890 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.240+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078019c0d0 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.241+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f077003c9f0 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.241+0000 7f077cff9640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0768038830 0x7f076803acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.241+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0780068740 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.241+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f077006c5b0 con 0x7f0780100990 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.241+0000 7f077effd640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0768038830 0x7f076803acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.242+0000 7f077effd640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0768038830 0x7f076803acf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07740098c0 tx=0x7f0774006db0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:32.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.246+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f0770035080 con 0x7f0780100990 2024-09-13T15:27:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.498+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07800ff9b0 con 0x7f0780100990 2024-09-13T15:27:32.501 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.499+0000 7f077cff9640 1 -- 172.21.15.175:0/221135550 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f0770040020 con 0x7f0780100990 2024-09-13T15:27:32.502 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:32.502 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:32.502 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0768038830 msgr2=0x7f076803acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0768038830 0x7f076803acf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07740098c0 tx=0x7f0774006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 msgr2=0x7f078019b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f078019b570 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0770009230 tx=0x7f0770004e90 comp rx=0 tx=0).stop 2024-09-13T15:27:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 shutdown_connections 2024-09-13T15:27:32.503 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0768038830 0x7f076803acf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:32.504 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 --2- 172.21.15.175:0/221135550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f0780100990 0x7f078019b570 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:32.504 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 >> 172.21.15.175:0/221135550 conn(0x7f07800fa3b0 msgr2=0x7f078010a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:32.504 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 shutdown_connections 2024-09-13T15:27:32.504 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:32.501+0000 7f0785e52640 1 -- 172.21.15.175:0/221135550 wait complete. 2024-09-13T15:27:33.505 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:33.506 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:33.506 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:33.506 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:33.506 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:33 smithi090 ceph-mon[27811]: Deploying daemon node-exporter.smithi175 on smithi175 2024-09-13T15:27:33.506 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:33 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/221135550' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:34.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:34.179 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:34.392 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:34.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:34.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:27:36.955 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 -- 172.21.15.175:0/723922966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 msgr2=0x7f9540104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:36.955 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 --2- 172.21.15.175:0/723922966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540104a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9530009880 tx=0x7f953002a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:36.955 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 -- 172.21.15.175:0/723922966 shutdown_connections 2024-09-13T15:27:36.955 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 --2- 172.21.15.175:0/723922966 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540104a30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:36.955 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 -- 172.21.15.175:0/723922966 >> 172.21.15.175:0/723922966 conn(0x7f9540075490 msgr2=0x7f95400fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 -- 172.21.15.175:0/723922966 shutdown_connections 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 -- 172.21.15.175:0/723922966 wait complete. 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 Processor -- start 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.952+0000 7f9544e07640 1 -- start start 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f9544e07640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f9544e07640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9540068660 con 0x7f9540104650 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f953e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:36.956 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f953e59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:60684/0 (socket says 172.21.15.175:60684) 2024-09-13T15:27:36.957 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f953e59c640 1 -- 172.21.15.175:0/2605345988 learned_addr learned my addr 172.21.15.175:0/2605345988 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:36.957 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f953e59c640 1 -- 172.21.15.175:0/2605345988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9530009560 con 0x7f9540104650 2024-09-13T15:27:36.958 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.953+0000 7f953e59c640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540195de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9530009230 tx=0x7f9530004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:36.958 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.954+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9530005290 con 0x7f9540104650 2024-09-13T15:27:36.958 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.954+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9540196380 con 0x7f9540104650 2024-09-13T15:27:36.958 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.954+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f95300053f0 con 0x7f9540104650 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.954+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f953003c6b0 con 0x7f9540104650 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.954+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9540196800 con 0x7f9540104650 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.955+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95400ffd90 con 0x7f9540104650 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.955+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f95300035f0 con 0x7f9540104650 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.955+0000 7f95377fe640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f952c038830 0x7f952c03acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.955+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f953006c570 con 0x7f9540104650 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.955+0000 7f953dd9b640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f952c038830 0x7f952c03acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:36.959 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.955+0000 7f953dd9b640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f952c038830 0x7f952c03acf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f95280098c0 tx=0x7f9528006db0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:36.962 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:36.961+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f953003a360 con 0x7f9540104650 2024-09-13T15:27:37.201 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.200+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9540196b00 con 0x7f9540104650 2024-09-13T15:27:37.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.200+0000 7f95377fe640 1 -- 172.21.15.175:0/2605345988 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f953004a090 con 0x7f9540104650 2024-09-13T15:27:37.202 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:37.202 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:37.202 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:37.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.202+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f952c038830 msgr2=0x7f952c03acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:37.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f952c038830 0x7f952c03acf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f95280098c0 tx=0x7f9528006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:37.204 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 msgr2=0x7f9540195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540195de0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9530009230 tx=0x7f9530004e70 comp rx=0 tx=0).stop 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 shutdown_connections 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f952c038830 0x7f952c03acf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 --2- 172.21.15.175:0/2605345988 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9540104650 0x7f9540195de0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 >> 172.21.15.175:0/2605345988 conn(0x7f9540075490 msgr2=0x7f95400fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 shutdown_connections 2024-09-13T15:27:37.205 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:37.203+0000 7f9544e07640 1 -- 172.21.15.175:0/2605345988 wait complete. 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi175.otzykk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi175.otzykk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: Deploying daemon mgr.smithi175.otzykk on smithi175 2024-09-13T15:27:38.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:38 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2605345988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:38.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:38.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:39.205 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:39.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:39 smithi090 ceph-mon[27811]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:39.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:39 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:40.877 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 -- 172.21.15.175:0/4098249970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 msgr2=0x7f0188105b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:40.877 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 --2- 172.21.15.175:0/4098249970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188105b80 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f017c009880 tx=0x7f017c02a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:40.877 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 -- 172.21.15.175:0/4098249970 shutdown_connections 2024-09-13T15:27:40.877 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 --2- 172.21.15.175:0/4098249970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188105b80 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 -- 172.21.15.175:0/4098249970 >> 172.21.15.175:0/4098249970 conn(0x7f0188076500 msgr2=0x7f0188076910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 -- 172.21.15.175:0/4098249970 shutdown_connections 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.870+0000 7f018ee0c640 1 -- 172.21.15.175:0/4098249970 wait complete. 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.871+0000 7f018ee0c640 1 Processor -- start 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.871+0000 7f018ee0c640 1 -- start start 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018ee0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188194ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018ee0c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0188195420 con 0x7f01881057a0 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018cba8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188194ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018cba8640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188194ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:45316/0 (socket says 172.21.15.175:45316) 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018cba8640 1 -- 172.21.15.175:0/1564222813 learned_addr learned my addr 172.21.15.175:0/1564222813 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018cba8640 1 -- 172.21.15.175:0/1564222813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c009560 con 0x7f01881057a0 2024-09-13T15:27:40.878 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f018cba8640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188194ee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f017c009230 tx=0x7f017c004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.872+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f017c0053e0 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.873+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188195680 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.873+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f017c005540 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.873+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f017c03c7e0 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.873+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0188195b00 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.874+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f017c0035f0 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.874+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0188100e70 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.874+0000 7f0185ffb640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0178038830 0x7f017803acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.874+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f017c06c6d0 con 0x7f01881057a0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.874+0000 7f0187fff640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0178038830 0x7f017803acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:40.879 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.874+0000 7f0187fff640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0178038830 0x7f017803acf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f01740098c0 tx=0x7f0174006db0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:40.880 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:40.879+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f017c03a540 con 0x7f01881057a0 2024-09-13T15:27:41.116 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.114+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01881099d0 con 0x7f01881057a0 2024-09-13T15:27:41.116 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.114+0000 7f0185ffb640 1 -- 172.21.15.175:0/1564222813 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f017c04a090 con 0x7f01881057a0 2024-09-13T15:27:41.116 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:41.116 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:41.117 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:41.118 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.116+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0178038830 msgr2=0x7f017803acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:41.118 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.116+0000 7f018ee0c640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0178038830 0x7f017803acf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f01740098c0 tx=0x7f0174006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:41.118 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.116+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 msgr2=0x7f0188194ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:41.118 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.116+0000 7f018ee0c640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188194ee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f017c009230 tx=0x7f017c004e70 comp rx=0 tx=0).stop 2024-09-13T15:27:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.117+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 shutdown_connections 2024-09-13T15:27:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.117+0000 7f018ee0c640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f0178038830 0x7f017803acf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.117+0000 7f018ee0c640 1 --2- 172.21.15.175:0/1564222813 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f01881057a0 0x7f0188194ee0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.117+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 >> 172.21.15.175:0/1564222813 conn(0x7f0188076500 msgr2=0x7f0188101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.117+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 shutdown_connections 2024-09-13T15:27:41.119 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:41.117+0000 7f018ee0c640 1 -- 172.21.15.175:0/1564222813 wait complete. 2024-09-13T15:27:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:41 smithi090 ceph-mon[27811]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:41 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/1564222813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:42.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:43.000 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:43.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T15:27:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:42 smithi090 ceph-mon[27811]: Deploying daemon mon.smithi175 on smithi175 2024-09-13T15:27:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:43 smithi090 ceph-mon[27811]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:44.310 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:27:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:45 smithi090 ceph-mon[27811]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.316+0000 7f02fa0cd640 1 -- 172.21.15.175:0/4110402818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 msgr2=0x7f02f4104b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.316+0000 7f02fa0cd640 1 --2- 172.21.15.175:0/4110402818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4104b70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009880 tx=0x7f02e402a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.316+0000 7f02fa0cd640 1 -- 172.21.15.175:0/4110402818 shutdown_connections 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.316+0000 7f02fa0cd640 1 --2- 172.21.15.175:0/4110402818 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4104b70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.316+0000 7f02fa0cd640 1 -- 172.21.15.175:0/4110402818 >> 172.21.15.175:0/4110402818 conn(0x7f02f40fa4b0 msgr2=0x7f02f40fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.317+0000 7f02fa0cd640 1 -- 172.21.15.175:0/4110402818 shutdown_connections 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.317+0000 7f02fa0cd640 1 -- 172.21.15.175:0/4110402818 wait complete. 2024-09-13T15:27:46.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.317+0000 7f02fa0cd640 1 Processor -- start 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.318+0000 7f02fa0cd640 1 -- start start 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.318+0000 7f02fa0cd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4193f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.318+0000 7f02fa0cd640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f4194470 con 0x7f02f4104790 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.318+0000 7f02f37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4193f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.318+0000 7f02f37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4193f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.175:45368/0 (socket says 172.21.15.175:45368) 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.318+0000 7f02f37fe640 1 -- 172.21.15.175:0/2866040233 learned_addr learned my addr 172.21.15.175:0/2866040233 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.319+0000 7f02f37fe640 1 -- 172.21.15.175:0/2866040233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4009560 con 0x7f02f4104790 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.319+0000 7f02f37fe640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4193f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009230 tx=0x7f02e4004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.319+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e40053e0 con 0x7f02f4104790 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.319+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f41946d0 con 0x7f02f4104790 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.320+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f4194b50 con 0x7f02f4104790 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.320+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f02e4005540 con 0x7f02f4104790 2024-09-13T15:27:46.578 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.320+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02e403c690 con 0x7f02f4104790 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.320+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 17) v1 ==== 45669+0+0 (secure 0 0 0) 0x7f02e403c7f0 con 0x7f02f4104790 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.321+0000 7f02f0ff9640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f02dc038830 0x7f02dc03acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.321+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f40ffed0 con 0x7f02f4104790 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.321+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f02e406c580 con 0x7f02f4104790 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.321+0000 7f02f2ffd640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f02dc038830 0x7f02dc03acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.321+0000 7f02f2ffd640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f02dc038830 0x7f02dc03acf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098c0 tx=0x7f02e8006db0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:46.579 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.326+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f02e403a950 con 0x7f02f4104790 2024-09-13T15:27:46.628 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.627+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02f4108a10 con 0x7f02f4104790 2024-09-13T15:27:46.629 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.628+0000 7f02f0ff9640 1 -- 172.21.15.175:0/2866040233 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+744 (secure 0 0 0) 0x7f02e404a090 con 0x7f02f4104790 2024-09-13T15:27:46.629 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:46.629 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":1,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:25:02.614558Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-13T15:27:46.629 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 1 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f02dc038830 msgr2=0x7f02dc03acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f02dc038830 0x7f02dc03acf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f02e80098c0 tx=0x7f02e8006db0 comp rx=0 tx=0).stop 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 msgr2=0x7f02f4193f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4193f30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f02e4009230 tx=0x7f02e4004e70 comp rx=0 tx=0).stop 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 shutdown_connections 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f02dc038830 0x7f02dc03acf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:46.631 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 --2- 172.21.15.175:0/2866040233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f02f4104790 0x7f02f4193f30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:46.632 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 >> 172.21.15.175:0/2866040233 conn(0x7f02f40fa4b0 msgr2=0x7f02f40fa890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:46.632 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 shutdown_connections 2024-09-13T15:27:46.632 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:46.630+0000 7f02fa0cd640 1 -- 172.21.15.175:0/2866040233 wait complete. 2024-09-13T15:27:47.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:46 smithi090 ceph-mon[27811]: Standby manager daemon smithi175.otzykk started 2024-09-13T15:27:47.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:46 smithi090 ceph-mon[27811]: from='mgr.? 172.21.15.175:0/789611370' entity='mgr.smithi175.otzykk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi175.otzykk/crt"}]: dispatch 2024-09-13T15:27:47.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:46 smithi090 ceph-mon[27811]: from='mgr.? 172.21.15.175:0/789611370' entity='mgr.smithi175.otzykk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-13T15:27:47.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:46 smithi090 ceph-mon[27811]: from='mgr.? 172.21.15.175:0/789611370' entity='mgr.smithi175.otzykk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi175.otzykk/key"}]: dispatch 2024-09-13T15:27:47.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:46 smithi090 ceph-mon[27811]: from='mgr.? 172.21.15.175:0/789611370' entity='mgr.smithi175.otzykk' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-13T15:27:47.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2866040233' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:48.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-13T15:27:48.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-13T15:27:48.697 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:27:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: mon.smithi090 calling monitor election 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: mon.smithi175 calling monitor election 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:52.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: mon.smithi090 is new leader, mons smithi090,smithi175 in quorum (ranks 0,1) 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: monmap e2: 2 mons at {smithi090=[v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0],smithi175=[v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: fsmap 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: osdmap e5: 0 total, 0 up, 0 in 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: mgrmap e18: smithi090.fyvvpd(active, since 33s), standbys: smithi175.otzykk 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: overall HEALTH_OK 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:52.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:51 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:53.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:52 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:27:53.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:52 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:53.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:52 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:27:53.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:52 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:27:53.254 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 -- 172.21.15.175:0/3330220865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 msgr2=0x7efff00059c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:53.254 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 --2- 172.21.15.175:0/3330220865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7efff00059c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f00040ffc10 tx=0x7efffc02bce0 comp rx=0 tx=0).stop 2024-09-13T15:27:53.254 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 -- 172.21.15.175:0/3330220865 shutdown_connections 2024-09-13T15:27:53.255 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 --2- 172.21.15.175:0/3330220865 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7efff00059c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:53.255 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 -- 172.21.15.175:0/3330220865 >> 172.21.15.175:0/3330220865 conn(0x7f00040fa440 msgr2=0x7f00040fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:53.255 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 -- 172.21.15.175:0/3330220865 shutdown_connections 2024-09-13T15:27:53.255 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.253+0000 7f0008df5640 1 -- 172.21.15.175:0/3330220865 wait complete. 2024-09-13T15:27:53.256 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.254+0000 7f0008df5640 1 Processor -- start 2024-09-13T15:27:53.256 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.254+0000 7f0008df5640 1 -- start start 2024-09-13T15:27:53.256 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.254+0000 7f0008df5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7f0004195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:53.256 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.254+0000 7f0008df5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:53.256 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.254+0000 7f0008df5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0004069390 con 0x7efff0003530 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.254+0000 7f0008df5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000419ef60 con 0x7f0004196430 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:55648/0 (socket says 172.21.15.175:55648) 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 -- 172.21.15.175:0/3509213071 learned_addr learned my addr 172.21.15.175:0/3509213071 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 -- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 msgr2=0x7f0004196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f000259c640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7f0004195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 -- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 msgr2=0x7f0004196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0001d9b640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f000259c640 1 -- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 msgr2=0x7f0004196890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f000259c640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:53.257 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f000259c640 1 -- 172.21.15.175:0/3509213071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efffc011870 con 0x7efff0003530 2024-09-13T15:27:53.258 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f000259c640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7f0004195ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efffc000c00 tx=0x7efffc02c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efffc005050 con 0x7efff0003530 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.255+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000419f160 con 0x7efff0003530 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.256+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efffc005640 con 0x7efff0003530 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.256+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efffc0048b0 con 0x7efff0003530 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.256+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000419f6b0 con 0x7efff0003530 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.256+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00040ffec0 con 0x7efff0003530 2024-09-13T15:27:53.259 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.257+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7efffc0037f0 con 0x7efff0003530 2024-09-13T15:27:53.260 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.258+0000 7efffb7fe640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7effe406ccd0 0x7effe406f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:53.260 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.258+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7efffc032070 con 0x7efff0003530 2024-09-13T15:27:53.260 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.258+0000 7f0001d9b640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7effe406ccd0 0x7effe406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:53.260 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.258+0000 7f0001d9b640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7effe406ccd0 0x7effe406f190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7effec009100 tx=0x7effec008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:53.263 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.262+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7efffc077800 con 0x7efff0003530 2024-09-13T15:27:53.494 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.493+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0004107650 con 0x7efff0003530 2024-09-13T15:27:53.495 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.494+0000 7efffb7fe640 1 -- 172.21.15.175:0/3509213071 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1022 (secure 0 0 0) 0x7efffc045410 con 0x7efff0003530 2024-09-13T15:27:53.495 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:27:53.495 INFO:teuthology.orchestra.run.smithi175.stdout:{"epoch":2,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","modified":"2024-09-13T15:27:46.726713Z","created":"2024-09-13T15:25:02.614558Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"smithi090","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:3300","nonce":0},{"type":"v1","addr":"172.21.15.90:6789","nonce":0}]},"addr":"172.21.15.90:6789/0","public_addr":"172.21.15.90:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi175","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:3300","nonce":0},{"type":"v1","addr":"172.21.15.175:6789","nonce":0}]},"addr":"172.21.15.175:6789/0","public_addr":"172.21.15.175:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-13T15:27:53.496 INFO:teuthology.orchestra.run.smithi175.stderr:dumped monmap epoch 2 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7effe406ccd0 msgr2=0x7effe406f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7effe406ccd0 0x7effe406f190 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7effec009100 tx=0x7effec008040 comp rx=0 tx=0).stop 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 msgr2=0x7f0004195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7f0004195ef0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efffc000c00 tx=0x7efffc02c800 comp rx=0 tx=0).stop 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 shutdown_connections 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7effe406ccd0 0x7effe406f190 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:53.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0004196430 0x7f0004196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:53.498 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 --2- 172.21.15.175:0/3509213071 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efff0003530 0x7f0004195ef0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:53.498 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 >> 172.21.15.175:0/3509213071 conn(0x7f00040fa440 msgr2=0x7f000418cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:53.498 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 shutdown_connections 2024-09-13T15:27:53.498 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:27:53.496+0000 7f0008df5640 1 -- 172.21.15.175:0/3509213071 wait complete. 2024-09-13T15:27:54.092 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:53 smithi175 ceph-mon[36128]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:54.092 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:53 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/3509213071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:54.092 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:54.112 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-13T15:27:54.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-13T15:27:54.141 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:53 smithi090 ceph-mon[27811]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:54.141 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:53 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/3509213071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-13T15:27:54.141 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:54.331 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:27:55.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: Updating smithi090:/etc/ceph/ceph.conf 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: Updating smithi175:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: Updating smithi090:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: Reconfiguring mon.smithi090 (unknown last config time)... 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:55.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:55 smithi090 ceph-mon[27811]: Reconfiguring daemon mon.smithi090 on smithi090 2024-09-13T15:27:55.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: Updating smithi090:/etc/ceph/ceph.conf 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: Updating smithi175:/etc/ceph/ceph.conf 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: Updating smithi175:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: Updating smithi090:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:55.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: Reconfiguring mon.smithi090 (unknown last config time)... 2024-09-13T15:27:55.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T15:27:55.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T15:27:55.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:55.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:55 smithi175 ceph-mon[36128]: Reconfiguring daemon mon.smithi090 on smithi090 2024-09-13T15:27:57.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1536977749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 msgr2=0x7f5ce8104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:57.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1536977749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce8104ae0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009880 tx=0x7f5cd802a5d0 comp rx=0 tx=0).stop 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1536977749 shutdown_connections 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1536977749 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce8104ae0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1536977749 >> 172.21.15.90:0/1536977749 conn(0x7f5ce80fa440 msgr2=0x7f5ce80fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1536977749 shutdown_connections 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.008+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1536977749 wait complete. 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5cedb6b640 1 Processor -- start 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5cedb6b640 1 -- start start 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5cedb6b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce80fa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5cedb6b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ce80fae30 0x7f5ce80fb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5cedb6b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce80fc7e0 con 0x7f5ce8104700 2024-09-13T15:27:57.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5cedb6b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81a1230 con 0x7f5ce80fae30 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5ce77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce80fa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5ce77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce80fa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34968/0 (socket says 172.21.15.90:34968) 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.009+0000 7f5ce77fe640 1 -- 172.21.15.90:0/1468210284 learned_addr learned my addr 172.21.15.90:0/1468210284 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce77fe640 1 -- 172.21.15.90:0/1468210284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ce80fae30 msgr2=0x7f5ce80fb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce77fe640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ce80fae30 0x7f5ce80fb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce77fe640 1 -- 172.21.15.90:0/1468210284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd8009560 con 0x7f5ce8104700 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce77fe640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce80fa8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8005ab0 tx=0x7f5cd8004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cd8005460 con 0x7f5ce8104700 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce81a13d0 con 0x7f5ce8104700 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5cd80055c0 con 0x7f5ce8104700 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5cd803c8a0 con 0x7f5ce8104700 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.010+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce81a18f0 con 0x7f5ce8104700 2024-09-13T15:27:57.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.011+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce80ffe40 con 0x7f5ce8104700 2024-09-13T15:27:57.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.012+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f5cd80035f0 con 0x7f5ce8104700 2024-09-13T15:27:57.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.012+0000 7f5ce4ff9640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5cd406ccd0 0x7f5cd406f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:27:57.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.012+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f5cd80ac2b0 con 0x7f5ce8104700 2024-09-13T15:27:57.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.012+0000 7f5ce6ffd640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5cd406ccd0 0x7f5cd406f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:27:57.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.013+0000 7f5ce6ffd640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5cd406ccd0 0x7f5cd406f190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0006fd0 tx=0x7f5cd0006d40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:27:57.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.020+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f5cd80776d0 con 0x7f5ce8104700 2024-09-13T15:27:57.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.202+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f5ce8104ae0 con 0x7f5ce8104700 2024-09-13T15:27:57.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.202+0000 7f5ce4ff9640 1 -- 172.21.15.90:0/1468210284 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f5cd80aba70 con 0x7f5ce8104700 2024-09-13T15:27:57.204 INFO:teuthology.orchestra.run.smithi090.stdout:# minimal ceph.conf for 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:27:57.204 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2024-09-13T15:27:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T15:27:57.204 INFO:teuthology.orchestra.run.smithi090.stdout: mon_host = [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5cd406ccd0 msgr2=0x7f5cd406f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5cd406ccd0 0x7f5cd406f190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0006fd0 tx=0x7f5cd0006d40 comp rx=0 tx=0).stop 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 msgr2=0x7f5ce80fa8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce80fa8f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8005ab0 tx=0x7f5cd8004e30 comp rx=0 tx=0).stop 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 shutdown_connections 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5cd406ccd0 0x7f5cd406f190 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.204+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5ce80fae30 0x7f5ce80fb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.205+0000 7f5cedb6b640 1 --2- 172.21.15.90:0/1468210284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5ce8104700 0x7f5ce80fa8f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:27:57.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.205+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 >> 172.21.15.90:0/1468210284 conn(0x7f5ce80fa440 msgr2=0x7f5ce818dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:27:57.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.205+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 shutdown_connections 2024-09-13T15:27:57.207 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:27:57.205+0000 7f5cedb6b640 1 -- 172.21.15.90:0/1468210284 wait complete. 2024-09-13T15:27:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: Reconfiguring mgr.smithi090.fyvvpd (unknown last config time)... 2024-09-13T15:27:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.fyvvpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T15:27:58.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T15:27:58.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:58.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: Reconfiguring daemon mgr.smithi090.fyvvpd on smithi090 2024-09-13T15:27:58.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:57 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1468210284' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: Reconfiguring mgr.smithi090.fyvvpd (unknown last config time)... 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi090.fyvvpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: Reconfiguring daemon mgr.smithi090.fyvvpd on smithi090 2024-09-13T15:27:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:57 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1468210284' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:27:58.368 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-13T15:27:58.368 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:27:58.368 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T15:27:58.436 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:27:58.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T15:27:58.501 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:27:58.501 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.conf 2024-09-13T15:27:58.535 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:27:58.535 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-13T15:27:58.605 INFO:tasks.cephadm:Deploying OSDs... 2024-09-13T15:27:58.606 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:27:58.606 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T15:27:58.622 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T15:27:58.622 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-13 15:27:12.159954466 +0000 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-13 15:21:52.409426955 +0000 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-13 15:21:52.409426955 +0000 2024-09-13T15:27:58.678 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-13 15:21:52.409426955 +0000 2024-09-13T15:27:58.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T15:27:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-13T15:27:58.741 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-13T15:27:58.742 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000264254 s, 1.9 MB/s 2024-09-13T15:27:58.742 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T15:27:58.799 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2024-09-13T15:27:58.854 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-13 15:27:12.160954450 +0000 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-13 15:21:52.743421720 +0000 2024-09-13T15:27:58.855 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-13 15:21:52.743421720 +0000 2024-09-13T15:27:58.856 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-13 15:21:52.743421720 +0000 2024-09-13T15:27:58.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T15:27:58.919 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-13T15:27:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-13T15:27:58.920 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000177509 s, 2.9 MB/s 2024-09-13T15:27:58.921 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T15:27:58.977 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2024-09-13T15:27:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-13 15:27:12.160954450 +0000 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-13 15:21:53.076416500 +0000 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-13 15:21:53.076416500 +0000 2024-09-13T15:27:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-13 15:21:53.076416500 +0000 2024-09-13T15:27:59.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T15:27:59.106 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-13T15:27:59.382 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-13T15:27:59.383 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000227456 s, 2.3 MB/s 2024-09-13T15:27:59.384 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T15:27:59.446 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2024-09-13T15:27:59.502 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2024-09-13 15:27:12.160954450 +0000 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2024-09-13 15:21:53.397411469 +0000 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2024-09-13 15:21:53.397411469 +0000 2024-09-13T15:27:59.503 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: 2024-09-13 15:21:53.397411469 +0000 2024-09-13T15:27:59.504 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T15:27:59.568 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2024-09-13T15:27:59.569 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2024-09-13T15:27:59.569 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000217818 s, 2.4 MB/s 2024-09-13T15:27:59.570 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T15:27:59.626 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:27:59.626 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T15:27:59.646 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T15:27:59.647 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 2024-09-13T15:27:59.705 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-13 15:27:44.632629556 +0000 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-13 15:21:52.125933638 +0000 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-13 15:21:52.125933638 +0000 2024-09-13T15:27:59.706 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-13 15:21:52.125933638 +0000 2024-09-13T15:27:59.706 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T15:27:59.774 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-13T15:27:59.774 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-13T15:27:59.774 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000239685 s, 2.1 MB/s 2024-09-13T15:27:59.775 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T15:27:59.836 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-13 15:27:44.632629556 +0000 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-13 15:21:52.463928349 +0000 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-13 15:21:52.463928349 +0000 2024-09-13T15:27:59.895 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-13 15:21:52.463928349 +0000 2024-09-13T15:27:59.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T15:27:59.963 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-13T15:27:59.963 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-13T15:27:59.963 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000192617 s, 2.7 MB/s 2024-09-13T15:27:59.964 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T15:28:00.024 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 2024-09-13T15:28:00.081 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-13 15:27:44.633629540 +0000 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-13 15:21:52.797923122 +0000 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-13 15:21:52.797923122 +0000 2024-09-13T15:28:00.082 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-13 15:21:52.797923122 +0000 2024-09-13T15:28:00.083 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T15:28:00.146 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:59 smithi175 ceph-mon[36128]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:00.146 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:00.146 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:00.146 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-13T15:28:00.146 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:27:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:00.148 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-13T15:28:00.148 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-13T15:28:00.148 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000195235 s, 2.6 MB/s 2024-09-13T15:28:00.150 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T15:28:00.201 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:59 smithi090 ceph-mon[27811]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:00.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:00.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:00.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi090", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-13T15:28:00.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:27:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:00.209 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 2024-09-13T15:28:00.266 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-09-13 15:27:44.633629540 +0000 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-13 15:21:53.123918019 +0000 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-13 15:21:53.123918019 +0000 2024-09-13T15:28:00.267 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-09-13 15:21:53.123918019 +0000 2024-09-13T15:28:00.268 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T15:28:00.333 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-09-13T15:28:00.334 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-09-13T15:28:00.334 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000215459 s, 2.4 MB/s 2024-09-13T15:28:00.335 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T15:28:00.393 INFO:tasks.cephadm:Deploying osd.0 on smithi090 with /dev/vg_nvme/lv_4... 2024-09-13T15:28:00.394 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-13T15:28:00.855 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:00 smithi090 ceph-mon[27811]: Reconfiguring ceph-exporter.smithi090 (monmap changed)... 2024-09-13T15:28:00.856 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:00 smithi090 ceph-mon[27811]: Reconfiguring daemon ceph-exporter.smithi090 on smithi090 2024-09-13T15:28:00.856 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:00 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:00.856 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:00 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:00.856 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:00 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-13T15:28:00.856 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:00 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:01.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:00 smithi175 ceph-mon[36128]: Reconfiguring ceph-exporter.smithi090 (monmap changed)... 2024-09-13T15:28:01.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:00 smithi175 ceph-mon[36128]: Reconfiguring daemon ceph-exporter.smithi090 on smithi090 2024-09-13T15:28:01.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:00 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:01.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:00 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:01.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:00 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi090", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-13T15:28:01.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:00 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:01.571 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:28:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:01 smithi090 ceph-mon[27811]: Reconfiguring crash.smithi090 (monmap changed)... 2024-09-13T15:28:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:01 smithi090 ceph-mon[27811]: Reconfiguring daemon crash.smithi090 on smithi090 2024-09-13T15:28:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:01 smithi090 ceph-mon[27811]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:01 smithi175 ceph-mon[36128]: Reconfiguring crash.smithi090 (monmap changed)... 2024-09-13T15:28:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:01 smithi175 ceph-mon[36128]: Reconfiguring daemon crash.smithi090 on smithi090 2024-09-13T15:28:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:01 smithi175 ceph-mon[36128]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:03.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:02 smithi090 ceph-mon[27811]: Reconfiguring alertmanager.smithi090 (dependencies changed)... 2024-09-13T15:28:03.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:02 smithi090 ceph-mon[27811]: Reconfiguring daemon alertmanager.smithi090 on smithi090 2024-09-13T15:28:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:02 smithi175 ceph-mon[36128]: Reconfiguring alertmanager.smithi090 (dependencies changed)... 2024-09-13T15:28:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:02 smithi175 ceph-mon[36128]: Reconfiguring daemon alertmanager.smithi090 on smithi090 2024-09-13T15:28:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:03 smithi090 ceph-mon[27811]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:04.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:03 smithi175 ceph-mon[36128]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:04.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:04.503 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-13T15:28:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-13T15:28:04.609 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-09-13T15:28:04.609 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-09-13T15:28:04.609 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116543 s, 900 MB/s 2024-09-13T15:28:04.609 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-09-13T15:28:06.247 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:05 smithi090 ceph-mon[27811]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:05 smithi175 ceph-mon[36128]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:06.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_4 2024-09-13T15:28:06.774 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:28:07.954 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:07 smithi090 ceph-mon[27811]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:07 smithi175 ceph-mon[36128]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:09.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 -- 172.21.15.90:0/3688035970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df40744e0 msgr2=0x7f4df4072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/3688035970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df40744e0 0x7f4df4072940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009880 tx=0x7f4df002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 -- 172.21.15.90:0/3688035970 shutdown_connections 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/3688035970 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df4072f10 0x7f4df4073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/3688035970 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df40744e0 0x7f4df4072940 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 -- 172.21.15.90:0/3688035970 >> 172.21.15.90:0/3688035970 conn(0x7f4df40fb320 msgr2=0x7f4df40fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 -- 172.21.15.90:0/3688035970 shutdown_connections 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 -- 172.21.15.90:0/3688035970 wait complete. 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 Processor -- start 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 -- start start 2024-09-13T15:28:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.045+0000 7f4dfb48f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 0x7f4df4197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:28:09.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dfb48f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df40744e0 0x7f4df4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:28:09.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dfb48f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df41980d0 con 0x7f4df4072f10 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dfb48f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df4198240 con 0x7f4df40744e0 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 0x7f4df4197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 0x7f4df4197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:56318/0 (socket says 172.21.15.90:56318) 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 -- 172.21.15.90:0/2864484033 learned_addr learned my addr 172.21.15.90:0/2864484033 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df8a2a640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df40744e0 0x7f4df4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 -- 172.21.15.90:0/2864484033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df40744e0 msgr2=0x7f4df4197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df40744e0 0x7f4df4197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 -- 172.21.15.90:0/2864484033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0009560 con 0x7f4df4072f10 2024-09-13T15:28:09.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df8a2a640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df40744e0 0x7f4df4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:28:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4df922b640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 0x7f4df4197650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009230 tx=0x7f4df0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:28:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df0005560 con 0x7f4df4072f10 2024-09-13T15:28:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4df0003a80 con 0x7f4df4072f10 2024-09-13T15:28:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df419ccc0 con 0x7f4df4072f10 2024-09-13T15:28:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4df003c9e0 con 0x7f4df4072f10 2024-09-13T15:28:09.052 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.046+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df419d160 con 0x7f4df4072f10 2024-09-13T15:28:09.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.048+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df410a7a0 con 0x7f4df4072f10 2024-09-13T15:28:09.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.048+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f4df00035f0 con 0x7f4df4072f10 2024-09-13T15:28:09.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.048+0000 7f4dea7fc640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f4de006cbc0 0x7f4de006f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:28:09.053 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.048+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1370+0+0 (secure 0 0 0) 0x7f4df00ac400 con 0x7f4df4072f10 2024-09-13T15:28:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.048+0000 7f4df8a2a640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f4de006cbc0 0x7f4de006f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:28:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.049+0000 7f4df8a2a640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f4de006cbc0 0x7f4de006f080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4de40058c0 tx=0x7f4de4005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:28:09.054 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.053+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f4df00778a0 con 0x7f4df4072f10 2024-09-13T15:28:09.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:09.240+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4df4107090 con 0x7f4de006cbc0 2024-09-13T15:28:10.126 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:09 smithi090 ceph-mon[27811]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:10.126 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:28:10.127 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:28:10.127 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:09 smithi175 ceph-mon[36128]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:28:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:28:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:10.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:10 smithi090 ceph-mon[27811]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:28:10.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:10 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:10.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:10 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:10.962 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:10 smithi090 ceph-mon[27811]: Reconfiguring grafana.smithi090 (dependencies changed)... 2024-09-13T15:28:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:10 smithi175 ceph-mon[36128]: from='client.14254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:28:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:10 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:11.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:10 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:11.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:10 smithi175 ceph-mon[36128]: Reconfiguring grafana.smithi090 (dependencies changed)... 2024-09-13T15:28:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:11 smithi090 ceph-mon[27811]: Reconfiguring daemon grafana.smithi090 on smithi090 2024-09-13T15:28:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:11 smithi090 ceph-mon[27811]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:11 smithi175 ceph-mon[36128]: Reconfiguring daemon grafana.smithi090 on smithi090 2024-09-13T15:28:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:11 smithi175 ceph-mon[36128]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:14.220 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:13 smithi090 ceph-mon[27811]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:13 smithi175 ceph-mon[36128]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:15 smithi090 ceph-mon[27811]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:16.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:15 smithi090 ceph-mon[27811]: Reconfiguring prometheus.smithi090 (dependencies changed)... 2024-09-13T15:28:16.068 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:15 smithi175 ceph-mon[36128]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:16.068 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:16.068 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:16.068 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:15 smithi175 ceph-mon[36128]: Reconfiguring prometheus.smithi090 (dependencies changed)... 2024-09-13T15:28:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:16 smithi090 ceph-mon[27811]: Reconfiguring daemon prometheus.smithi090 on smithi090 2024-09-13T15:28:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:16 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2926247315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "358fe0ea-a41d-4789-903d-7e4613b4a0da"}]: dispatch 2024-09-13T15:28:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:16 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2926247315' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "358fe0ea-a41d-4789-903d-7e4613b4a0da"}]': finished 2024-09-13T15:28:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:16 smithi090 ceph-mon[27811]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T15:28:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:16 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/70722215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:28:17.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:16 smithi175 ceph-mon[36128]: Reconfiguring daemon prometheus.smithi090 on smithi090 2024-09-13T15:28:17.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:16 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2926247315' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "358fe0ea-a41d-4789-903d-7e4613b4a0da"}]: dispatch 2024-09-13T15:28:17.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:16 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2926247315' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "358fe0ea-a41d-4789-903d-7e4613b4a0da"}]': finished 2024-09-13T15:28:17.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:16 smithi175 ceph-mon[36128]: osdmap e6: 1 total, 0 up, 1 in 2024-09-13T15:28:17.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:17.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:16 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/70722215' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:28:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:17 smithi090 ceph-mon[27811]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:17 smithi175 ceph-mon[36128]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:19.244 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:19.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:19 smithi090 ceph-mon[27811]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:19 smithi175 ceph-mon[36128]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:21.994 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:21 smithi090 ceph-mon[27811]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:21 smithi175 ceph-mon[36128]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:23.981 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:23 smithi090 ceph-mon[27811]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:23 smithi175 ceph-mon[36128]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: Reconfiguring ceph-exporter.smithi175 (monmap changed)... 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi175", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:25.964 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:25 smithi090 ceph-mon[27811]: Reconfiguring daemon ceph-exporter.smithi175 on smithi175 2024-09-13T15:28:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: Reconfiguring ceph-exporter.smithi175 (monmap changed)... 2024-09-13T15:28:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.ceph-exporter.smithi175", "caps": ["mon", "profile ceph-exporter", "mon", "allow r", "mgr", "allow r", "osd", "allow r"]}]: dispatch 2024-09-13T15:28:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:25 smithi175 ceph-mon[36128]: Reconfiguring daemon ceph-exporter.smithi175 on smithi175 2024-09-13T15:28:27.051 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: Reconfiguring crash.smithi175 (monmap changed)... 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi175", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: Reconfiguring daemon crash.smithi175 on smithi175 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: Deploying daemon osd.0 on smithi090 2024-09-13T15:28:27.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:26 smithi175 ceph-mon[36128]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:27.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:27.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: Reconfiguring crash.smithi175 (monmap changed)... 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi175", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: Reconfiguring daemon crash.smithi175 on smithi175 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: Deploying daemon osd.0 on smithi090 2024-09-13T15:28:27.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:26 smithi090 ceph-mon[27811]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:28.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:28.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: Reconfiguring mgr.smithi175.otzykk (monmap changed)... 2024-09-13T15:28:28.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi175.otzykk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T15:28:28.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T15:28:28.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:28.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:28 smithi090 ceph-mon[27811]: Reconfiguring daemon mgr.smithi175.otzykk on smithi175 2024-09-13T15:28:28.399 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:28.399 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:28.399 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: Reconfiguring mgr.smithi175.otzykk (monmap changed)... 2024-09-13T15:28:28.399 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi175.otzykk", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-13T15:28:28.400 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-13T15:28:28.400 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:28.400 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:28 smithi175 ceph-mon[36128]: Reconfiguring daemon mgr.smithi175.otzykk on smithi175 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: Reconfiguring mon.smithi175 (monmap changed)... 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: Reconfiguring daemon mon.smithi175 on smithi175 2024-09-13T15:28:29.658 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:29 smithi090 ceph-mon[27811]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:29.697 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:29.697 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:29.697 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: Reconfiguring mon.smithi175 (monmap changed)... 2024-09-13T15:28:29.698 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-13T15:28:29.698 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-13T15:28:29.698 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:29.698 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: Reconfiguring daemon mon.smithi175 on smithi175 2024-09-13T15:28:29.698 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:29 smithi175 ceph-mon[36128]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:31.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T15:28:31.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T15:28:31.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T15:28:31.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T15:28:31.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:28:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi090.front.sepia.ceph.com:9093"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi090.front.sepia.ceph.com:3000"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T15:28:31.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi090.front.sepia.ceph.com:9095"}]: dispatch 2024-09-13T15:28:31.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:31.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:28:32.090 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:31 smithi090 ceph-mon[27811]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:32.090 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.090 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.090 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.090 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:31 smithi175 ceph-mon[36128]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:32.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:32.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:34.211 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:33 smithi090 ceph-mon[27811]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:34.211 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:33 smithi175 ceph-mon[36128]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:34.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.516+0000 7f4dea7fc640 1 -- 172.21.15.90:0/2864484033 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4df4107090 con 0x7f4de006cbc0 2024-09-13T15:28:34.518 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 0 on host 'smithi090' 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f4de006cbc0 msgr2=0x7f4de006f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f4de006cbc0 0x7f4de006f080 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4de40058c0 tx=0x7f4de4005850 comp rx=0 tx=0).stop 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 msgr2=0x7f4df4197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 0x7f4df4197650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009230 tx=0x7f4df0004600 comp rx=0 tx=0).stop 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 shutdown_connections 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f4de006cbc0 0x7f4de006f080 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4df40744e0 0x7f4df4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 --2- 172.21.15.90:0/2864484033 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4df4072f10 0x7f4df4197650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:34.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.518+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 >> 172.21.15.90:0/2864484033 conn(0x7f4df40fb320 msgr2=0x7f4df4105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:28:34.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.519+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 shutdown_connections 2024-09-13T15:28:34.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:34.519+0000 7f4dfb48f640 1 -- 172.21.15.90:0/2864484033 wait complete. 2024-09-13T15:28:35.101 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:34 smithi090 ceph-mon[27811]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T15:28:35.101 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:35.101 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:35.122 DEBUG:teuthology.orchestra.run.smithi090:osd.0> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.0.service 2024-09-13T15:28:35.124 INFO:tasks.cephadm:Deploying osd.1 on smithi090 with /dev/vg_nvme/lv_3... 2024-09-13T15:28:35.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-13T15:28:35.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:34 smithi175 ceph-mon[36128]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-13T15:28:35.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:35.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:35.815 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:28:36.066 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:35 smithi090 ceph-mon[27811]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:36.067 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:35 smithi090 ceph-mon[27811]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T15:28:36.067 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:35 smithi090 ceph-mon[27811]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T15:28:36.067 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:36.067 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:35 smithi090 ceph-mon[27811]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:28:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:35 smithi175 ceph-mon[36128]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:35 smithi175 ceph-mon[36128]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-13T15:28:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:35 smithi175 ceph-mon[36128]: osdmap e7: 1 total, 0 up, 1 in 2024-09-13T15:28:36.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:36.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:35 smithi175 ceph-mon[36128]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:28:36.345 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0[45757]: 2024-09-13T15:28:36.227+0000 7fb73bc86640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T15:28:36.346 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0[45757]: 2024-09-13T15:28:36.229+0000 7fb733290640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:28:37.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-mon[27811]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:28:37.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-mon[27811]: osdmap e8: 1 total, 0 up, 1 in 2024-09-13T15:28:37.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:37.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:37.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-mon[27811]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' 2024-09-13T15:28:37.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:37.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:36 smithi175 ceph-mon[36128]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:28:37.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:36 smithi175 ceph-mon[36128]: osdmap e8: 1 total, 0 up, 1 in 2024-09-13T15:28:37.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:36 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:37.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:36 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:37.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:36 smithi175 ceph-mon[36128]: from='osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213]' entity='osd.0' 2024-09-13T15:28:37.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:36 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:37.957 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:37 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:28:37.957 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:37 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:28:37.957 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:37 smithi090 ceph-mon[27811]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:37.957 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:37 smithi090 ceph-mon[27811]: osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] boot 2024-09-13T15:28:37.957 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:37 smithi090 ceph-mon[27811]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T15:28:37.958 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:38.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:37 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:28:38.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:37 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:28:38.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:37 smithi175 ceph-mon[36128]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-13T15:28:38.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:37 smithi175 ceph-mon[36128]: osd.0 [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] boot 2024-09-13T15:28:38.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:37 smithi175 ceph-mon[36128]: osdmap e9: 1 total, 1 up, 1 in 2024-09-13T15:28:38.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-13T15:28:38.797 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-13T15:28:38.797 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-13T15:28:38.797 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-09-13T15:28:38.797 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-09-13T15:28:38.797 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0158917 s, 660 MB/s 2024-09-13T15:28:38.797 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-09-13T15:28:39.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:39 smithi090 ceph-mon[27811]: osdmap e10: 1 total, 1 up, 1 in 2024-09-13T15:28:39.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:39 smithi090 ceph-mon[27811]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:39.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:39 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:39.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:39 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:39.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_3 2024-09-13T15:28:39.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:39 smithi175 ceph-mon[36128]: osdmap e10: 1 total, 1 up, 1 in 2024-09-13T15:28:39.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:39 smithi175 ceph-mon[36128]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:39.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:39 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:39.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:39 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:39.600 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.164+0000 7f1bf8121640 1 -- 172.21.15.90:0/945023316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00febd0 msgr2=0x7f1bf00ff050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.164+0000 7f1bf8121640 1 --2- 172.21.15.90:0/945023316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00febd0 0x7f1bf00ff050 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1be0009880 tx=0x7f1be002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.164+0000 7f1bf8121640 1 -- 172.21.15.90:0/945023316 shutdown_connections 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.164+0000 7f1bf8121640 1 --2- 172.21.15.90:0/945023316 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00febd0 0x7f1bf00ff050 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.164+0000 7f1bf8121640 1 --2- 172.21.15.90:0/945023316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bf00fde60 0x7f1bf00fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.164+0000 7f1bf8121640 1 -- 172.21.15.90:0/945023316 >> 172.21.15.90:0/945023316 conn(0x7f1bf00f99b0 msgr2=0x7f1bf00fbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/945023316 shutdown_connections 2024-09-13T15:28:41.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/945023316 wait complete. 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 Processor -- start 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 -- start start 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 0x7f1bf01977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bf00febd0 0x7f1bf0197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0198270 con 0x7f1bf00fde60 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf8121640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf01983e0 con 0x7f1bf00febd0 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf5ebd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 0x7f1bf01977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf5ebd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 0x7f1bf01977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:49832/0 (socket says 172.21.15.90:49832) 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.165+0000 7f1bf5ebd640 1 -- 172.21.15.90:0/3938310787 learned_addr learned my addr 172.21.15.90:0/3938310787 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1bf5ebd640 1 -- 172.21.15.90:0/3938310787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bf00febd0 msgr2=0x7f1bf0197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1bf5ebd640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bf00febd0 0x7f1bf0197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:28:41.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1bf5ebd640 1 -- 172.21.15.90:0/3938310787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0009560 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1bf5ebd640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 0x7f1bf01977f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00b420 tx=0x7f1bec00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bec011800 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1bec011df0 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf019cec0 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bec010e80 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.166+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf019d410 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.167+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf010a840 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.167+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f1bec011960 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.168+0000 7f1be6ffd640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1bdc06cbd0 0x7f1bdc06f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.168+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1982+0+0 (secure 0 0 0) 0x7f1bec08ad20 con 0x7f1bf00fde60 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.168+0000 7f1bf56bc640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1bdc06cbd0 0x7f1bdc06f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:28:41.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.168+0000 7f1bf56bc640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1bdc06cbd0 0x7f1bdc06f090 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1be002a9c0 tx=0x7f1be0000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:28:41.178 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.172+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1bec055f60 con 0x7f1bf00fde60 2024-09-13T15:28:41.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:28:41.357+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bf019d690 con 0x7f1bdc06cbd0 2024-09-13T15:28:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:41 smithi090 ceph-mon[27811]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:41 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:28:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:41 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:28:42.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:41 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:42.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:41 smithi175 ceph-mon[36128]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:42.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:41 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:28:42.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:41 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:28:42.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:41 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:42.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:42 smithi090 ceph-mon[27811]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:28:43.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:42 smithi175 ceph-mon[36128]: from='client.14272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:28:43.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:43 smithi175 ceph-mon[36128]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:43 smithi090 ceph-mon[27811]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi090 2024-09-13T15:28:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi090 to 11840M 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi090 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi090 to 11840M 2024-09-13T15:28:46.165 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:46.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:28:46.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:46.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:28:47.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/154489294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9260e3c1-8e73-4a12-8caf-f5f664fd7c5b"}]: dispatch 2024-09-13T15:28:47.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/154489294' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9260e3c1-8e73-4a12-8caf-f5f664fd7c5b"}]': finished 2024-09-13T15:28:47.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T15:28:47.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:28:47.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/542423500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:28:47.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:47.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/154489294' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9260e3c1-8e73-4a12-8caf-f5f664fd7c5b"}]: dispatch 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/154489294' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9260e3c1-8e73-4a12-8caf-f5f664fd7c5b"}]': finished 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: osdmap e11: 2 total, 1 up, 2 in 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/542423500' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:47.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:48.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:47 smithi090 ceph-mon[27811]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:47 smithi175 ceph-mon[36128]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:49.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:28:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:49 smithi090 ceph-mon[27811]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:50.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:28:50.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:49 smithi175 ceph-mon[36128]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:50.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:50.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:28:50.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:52.230 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:51 smithi090 ceph-mon[27811]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:51 smithi175 ceph-mon[36128]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:52.949 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:52 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T15:28:52.949 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:52 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:53.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:52 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-13T15:28:53.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:52 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:28:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:53 smithi090 ceph-mon[27811]: Deploying daemon osd.1 on smithi090 2024-09-13T15:28:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:53 smithi090 ceph-mon[27811]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:53 smithi175 ceph-mon[36128]: Deploying daemon osd.1 on smithi090 2024-09-13T15:28:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:53 smithi175 ceph-mon[36128]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:55.920 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:55 smithi090 ceph-mon[27811]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:55 smithi175 ceph-mon[36128]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:57 smithi090 ceph-mon[27811]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:28:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:57 smithi175 ceph-mon[36128]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:28:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:59.905 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:59 smithi090 ceph-mon[27811]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:28:59.905 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:28:59.905 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:28:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:59 smithi175 ceph-mon[36128]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:29:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:00.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:28:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:01.134 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:00 smithi090 ceph-mon[27811]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T15:29:01.164 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.163+0000 7f1be6ffd640 1 -- 172.21.15.90:0/3938310787 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1bf019d690 con 0x7f1bdc06cbd0 2024-09-13T15:29:01.164 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 1 on host 'smithi090' 2024-09-13T15:29:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1bdc06cbd0 msgr2=0x7f1bdc06f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1bdc06cbd0 0x7f1bdc06f090 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1be002a9c0 tx=0x7f1be0000bc0 comp rx=0 tx=0).stop 2024-09-13T15:29:01.166 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 msgr2=0x7f1bf01977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 0x7f1bf01977f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00b420 tx=0x7f1bec00b8e0 comp rx=0 tx=0).stop 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 shutdown_connections 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1bdc06cbd0 0x7f1bdc06f090 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bf00febd0 0x7f1bf0197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 --2- 172.21.15.90:0/3938310787 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1bf00fde60 0x7f1bf01977f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 >> 172.21.15.90:0/3938310787 conn(0x7f1bf00f99b0 msgr2=0x7f1bf0107260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 shutdown_connections 2024-09-13T15:29:01.167 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:01.165+0000 7f1bf8121640 1 -- 172.21.15.90:0/3938310787 wait complete. 2024-09-13T15:29:01.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:00 smithi175 ceph-mon[36128]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-13T15:29:01.771 DEBUG:teuthology.orchestra.run.smithi090:osd.1> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.1.service 2024-09-13T15:29:01.773 INFO:tasks.cephadm:Deploying osd.2 on smithi090 with /dev/vg_nvme/lv_2... 2024-09-13T15:29:01.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-13T15:29:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:29:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T15:29:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T15:29:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:29:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:02.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:02.002 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1[52224]: 2024-09-13T15:29:01.848+0000 7f8eec8c1640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T15:29:02.002 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:01 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1[52224]: 2024-09-13T15:29:01.850+0000 7f8ee86ea640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: osdmap e12: 2 total, 1 up, 2 in 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:02.450 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:29:03.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:02 smithi090 ceph-mon[27811]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:29:03.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:02 smithi090 ceph-mon[27811]: osdmap e13: 2 total, 1 up, 2 in 2024-09-13T15:29:03.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:02 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:03.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:02 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:02 smithi175 ceph-mon[36128]: from='osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:29:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:02 smithi175 ceph-mon[36128]: osdmap e13: 2 total, 1 up, 2 in 2024-09-13T15:29:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:02 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:02 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:29:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:29:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:29:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:04.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] boot 2024-09-13T15:29:04.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T15:29:04.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:04.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:29:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:29:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-13T15:29:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:04.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: osd.1 [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] boot 2024-09-13T15:29:04.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: osdmap e14: 2 total, 2 up, 2 in 2024-09-13T15:29:04.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-13T15:29:04.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:04.916 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:04 smithi090 ceph-mon[27811]: osdmap e15: 2 total, 2 up, 2 in 2024-09-13T15:29:05.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:04 smithi175 ceph-mon[36128]: osdmap e15: 2 total, 2 up, 2 in 2024-09-13T15:29:05.393 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-13T15:29:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-13T15:29:05.498 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-09-13T15:29:05.498 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-09-13T15:29:05.498 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139616 s, 751 MB/s 2024-09-13T15:29:05.499 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-09-13T15:29:06.124 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:05 smithi090 ceph-mon[27811]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:06.124 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:05 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:06.124 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:05 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:06.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_2 2024-09-13T15:29:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:05 smithi175 ceph-mon[36128]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:05 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:05 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:07.657 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:29:08.093 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:07 smithi090 ceph-mon[27811]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:08.268 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:07 smithi175 ceph-mon[36128]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 -- 172.21.15.90:0/74594238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 msgr2=0x7f59380feef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 --2- 172.21.15.90:0/74594238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f59380feef0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5928009880 tx=0x7f592802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 -- 172.21.15.90:0/74594238 shutdown_connections 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 --2- 172.21.15.90:0/74594238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 0x7f5938100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 --2- 172.21.15.90:0/74594238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f59380feef0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 -- 172.21.15.90:0/74594238 >> 172.21.15.90:0/74594238 conn(0x7f59380fa280 msgr2=0x7f59380fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 -- 172.21.15.90:0/74594238 shutdown_connections 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 -- 172.21.15.90:0/74594238 wait complete. 2024-09-13T15:29:09.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.829+0000 7f593fdaa640 1 Processor -- start 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593fdaa640 1 -- start start 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593fdaa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f5938102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593fdaa640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 0x7f59381010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593fdaa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938102fc0 con 0x7f59380feaf0 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593fdaa640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938101640 con 0x7f59380ffcf0 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f5938102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f5938102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55784/0 (socket says 172.21.15.90:55784) 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 -- 172.21.15.90:0/987534069 learned_addr learned my addr 172.21.15.90:0/987534069 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593d345640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 0x7f59381010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 -- 172.21.15.90:0/987534069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 msgr2=0x7f59381010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 0x7f59381010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:09.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 -- 172.21.15.90:0/987534069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928009560 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593d345640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 0x7f59381010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f593db46640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f5938102a80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5928006010 tx=0x7f5928004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.830+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5928005460 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.831+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59381018c0 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.831+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f59280055c0 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.831+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f592803c820 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.831+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5938101db0 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.832+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59380feef0 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.832+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f59280035f0 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.832+0000 7f592effd640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f592406cbd0 0x7f592406f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.833+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2414+0+0 (secure 0 0 0) 0x7f59280ac610 con 0x7f59380feaf0 2024-09-13T15:29:09.852 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.833+0000 7f593d345640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f592406cbd0 0x7f592406f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:09.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.833+0000 7f593d345640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f592406cbd0 0x7f592406f090 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5938102800 tx=0x7f5934005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:29:09.853 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:09.837+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f5928077750 con 0x7f59380feaf0 2024-09-13T15:29:10.033 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:10.031+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5938068340 con 0x7f592406cbd0 2024-09-13T15:29:10.131 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:10.131 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.131 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.131 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:10.131 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:10.131 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.132 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:10.132 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:10.132 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.132 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:10.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:10.222 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:10.222 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:11.067 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:10 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi090 2024-09-13T15:29:11.068 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:10 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi090 to 5920M 2024-09-13T15:29:11.068 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:10 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:29:11.068 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:10 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:29:11.068 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:10 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:10 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi090 2024-09-13T15:29:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:10 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi090 to 5920M 2024-09-13T15:29:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:10 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:29:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:10 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:29:11.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:10 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:12.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:11 smithi090 ceph-mon[27811]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:29:12.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:11 smithi090 ceph-mon[27811]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:12.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:12.170 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:11 smithi175 ceph-mon[36128]: from='client.14290 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:29:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:11 smithi175 ceph-mon[36128]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:12.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:14.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:13 smithi090 ceph-mon[27811]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:14.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:13 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/971261070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b19df8cf-75da-4134-badb-cdd4eb29c5db"}]: dispatch 2024-09-13T15:29:14.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:13 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/971261070' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b19df8cf-75da-4134-badb-cdd4eb29c5db"}]': finished 2024-09-13T15:29:14.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:13 smithi090 ceph-mon[27811]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T15:29:14.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:13 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:13 smithi175 ceph-mon[36128]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:13 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/971261070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b19df8cf-75da-4134-badb-cdd4eb29c5db"}]: dispatch 2024-09-13T15:29:14.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:13 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/971261070' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b19df8cf-75da-4134-badb-cdd4eb29c5db"}]': finished 2024-09-13T15:29:14.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:13 smithi175 ceph-mon[36128]: osdmap e16: 3 total, 2 up, 3 in 2024-09-13T15:29:14.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:13 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:15.016 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:14 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1745083967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:29:15.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:14 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1745083967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:29:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:15 smithi175 ceph-mon[36128]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:15 smithi090 ceph-mon[27811]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:18.239 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:17 smithi175 ceph-mon[36128]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:17 smithi090 ceph-mon[27811]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:18 smithi090 ceph-mon[27811]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:18 smithi175 ceph-mon[36128]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:20.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T15:29:20.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:20.206 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:19 smithi090 ceph-mon[27811]: Deploying daemon osd.2 on smithi090 2024-09-13T15:29:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-13T15:29:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:19 smithi175 ceph-mon[36128]: Deploying daemon osd.2 on smithi090 2024-09-13T15:29:21.154 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:20 smithi090 ceph-mon[27811]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:21.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:20 smithi175 ceph-mon[36128]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:23.850 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:23 smithi090 ceph-mon[27811]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:23 smithi175 ceph-mon[36128]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:25.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:25 smithi090 ceph-mon[27811]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:25.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:25.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:25 smithi175 ceph-mon[36128]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:26.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:27.843 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:27 smithi090 ceph-mon[27811]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:27 smithi175 ceph-mon[36128]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:28.146 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 2 on host 'smithi090' 2024-09-13T15:29:28.146 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.144+0000 7f592effd640 1 -- 172.21.15.90:0/987534069 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5938068340 con 0x7f592406cbd0 2024-09-13T15:29:28.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f592406cbd0 msgr2=0x7f592406f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:28.147 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f592406cbd0 0x7f592406f090 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5938102800 tx=0x7f5934005f90 comp rx=0 tx=0).stop 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 msgr2=0x7f5938102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f5938102a80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5928006010 tx=0x7f5928004c60 comp rx=0 tx=0).stop 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 shutdown_connections 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f592406cbd0 0x7f592406f090 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59380ffcf0 0x7f59381010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 --2- 172.21.15.90:0/987534069 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59380feaf0 0x7f5938102a80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 >> 172.21.15.90:0/987534069 conn(0x7f59380fa280 msgr2=0x7f5938067c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 shutdown_connections 2024-09-13T15:29:28.148 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:28.146+0000 7f593fdaa640 1 -- 172.21.15.90:0/987534069 wait complete. 2024-09-13T15:29:28.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:28 smithi090 ceph-mon[27811]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T15:29:28.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:28.750 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:28 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:28.771 DEBUG:teuthology.orchestra.run.smithi090:osd.2> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.2.service 2024-09-13T15:29:28.774 INFO:tasks.cephadm:Deploying osd.3 on smithi090 with /dev/vg_nvme/lv_1... 2024-09-13T15:29:28.774 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-13T15:29:29.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:28 smithi175 ceph-mon[36128]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-13T15:29:29.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:29.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:28 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:29.147 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:29:29.891 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:29 smithi090 ceph-mon[27811]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:29.891 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:29 smithi090 ceph-mon[27811]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T15:29:29.891 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:29 smithi090 ceph-mon[27811]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T15:29:29.891 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:29.891 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:29 smithi090 ceph-mon[27811]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:29:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:29 smithi175 ceph-mon[36128]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:29 smithi175 ceph-mon[36128]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-13T15:29:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:29 smithi175 ceph-mon[36128]: osdmap e17: 3 total, 2 up, 3 in 2024-09-13T15:29:30.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:30.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:29 smithi175 ceph-mon[36128]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:29:30.239 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2[58886]: 2024-09-13T15:29:30.046+0000 7fa615cdb640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T15:29:30.239 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2[58886]: 2024-09-13T15:29:30.050+0000 7fa60dafc640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:29:30.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-mon[27811]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:29:30.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-mon[27811]: osdmap e18: 3 total, 2 up, 3 in 2024-09-13T15:29:30.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:30.955 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:30.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-mon[27811]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' 2024-09-13T15:29:30.956 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:30 smithi175 ceph-mon[36128]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:29:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:30 smithi175 ceph-mon[36128]: osdmap e18: 3 total, 2 up, 3 in 2024-09-13T15:29:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:30 smithi175 ceph-mon[36128]: from='osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791]' entity='osd.2' 2024-09-13T15:29:31.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:31.890 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:31 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:29:31.890 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:31 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:29:31.890 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:31 smithi090 ceph-mon[27811]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:31.890 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:31 smithi090 ceph-mon[27811]: osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] boot 2024-09-13T15:29:31.890 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:31 smithi090 ceph-mon[27811]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T15:29:31.890 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:31 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:29:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:31 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:29:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:31 smithi175 ceph-mon[36128]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-13T15:29:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:31 smithi175 ceph-mon[36128]: osd.2 [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] boot 2024-09-13T15:29:32.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:31 smithi175 ceph-mon[36128]: osdmap e19: 3 total, 3 up, 3 in 2024-09-13T15:29:32.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-13T15:29:32.821 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-13T15:29:32.918 INFO:teuthology.orchestra.run.smithi090.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-13T15:29:32.935 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10+0 records in 2024-09-13T15:29:32.935 INFO:teuthology.orchestra.run.smithi090.stderr:10+0 records out 2024-09-13T15:29:32.935 INFO:teuthology.orchestra.run.smithi090.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149603 s, 701 MB/s 2024-09-13T15:29:32.935 INFO:teuthology.orchestra.run.smithi090.stderr:--> Zapping successful for: 2024-09-13T15:29:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:33.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:33.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:33.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T15:29:33.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T15:29:33.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T15:29:33.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: osdmap e20: 3 total, 3 up, 3 in 2024-09-13T15:29:33.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-13T15:29:33.544 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi090:vg_nvme/lv_1 2024-09-13T15:29:34.369 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:34.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:29:34.370 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:34 smithi090 ceph-mon[27811]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T15:29:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-13T15:29:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:34 smithi175 ceph-mon[36128]: osdmap e21: 3 total, 3 up, 3 in 2024-09-13T15:29:34.890 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:29:35.237 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:34 smithi090 sudo[60670]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T15:29:35.238 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:34 smithi090 sudo[60670]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:35.238 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:34 smithi090 sudo[60670]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:35.238 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:34 smithi090 sudo[60670]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:35.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:35 smithi090 ceph-mon[27811]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:35.501 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[61451]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T15:29:35.501 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[61451]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:35.501 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[61451]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:35.501 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[61451]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:35.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:35 smithi175 ceph-mon[36128]: pgmap v77: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:35.778 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63407]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T15:29:35.779 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63407]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:35.779 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63407]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:35.779 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63407]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:36.030 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T15:29:36.030 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:36.030 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63413]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:36.030 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:29:35 smithi090 sudo[63413]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:36.290 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 ceph-mon[27811]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T15:29:36.290 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:36.291 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63424]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T15:29:36.291 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63424]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:36.291 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63424]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:36.291 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63424]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:36.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:36 smithi175 ceph-mon[36128]: osdmap e22: 3 total, 3 up, 3 in 2024-09-13T15:29:36.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:36 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:36.542 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63439]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T15:29:36.542 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63439]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:36.542 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63439]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:36.542 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63439]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:36.803 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63443]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T15:29:36.803 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63443]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:36.803 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63443]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:36.804 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63443]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:37.120 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63448]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T15:29:37.120 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63448]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:37.120 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63448]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:37.120 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:36 smithi090 sudo[63448]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:37.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 -- 172.21.15.90:0/2537081144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 msgr2=0x7f3fac0fe250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:37.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2537081144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac0fe250 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c009880 tx=0x7f3f9c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:29:37.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 -- 172.21.15.90:0/2537081144 shutdown_connections 2024-09-13T15:29:37.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2537081144 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fac100e30 0x7f3fac1012b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:37.122 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2537081144 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac0fe250 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 -- 172.21.15.90:0/2537081144 >> 172.21.15.90:0/2537081144 conn(0x7f3fac0f99c0 msgr2=0x7f3fac0fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 -- 172.21.15.90:0/2537081144 shutdown_connections 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.118+0000 7f3fb266f640 1 -- 172.21.15.90:0/2537081144 wait complete. 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fb266f640 1 Processor -- start 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fb266f640 1 -- start start 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fb266f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fb266f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fac100e30 0x7f3fac195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fb266f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac195ed0 con 0x7f3fac0fde50 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fb266f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fac196040 con 0x7f3fac100e30 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fabfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fabfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:34142/0 (socket says 172.21.15.90:34142) 2024-09-13T15:29:37.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fabfff640 1 -- 172.21.15.90:0/2008766830 learned_addr learned my addr 172.21.15.90:0/2008766830 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.119+0000 7f3fa3fff640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fac100e30 0x7f3fac195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fabfff640 1 -- 172.21.15.90:0/2008766830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fac100e30 msgr2=0x7f3fac195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fabfff640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fac100e30 0x7f3fac195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fabfff640 1 -- 172.21.15.90:0/2008766830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c009560 con 0x7f3fac0fde50 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fabfff640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac195450 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c006010 tx=0x7f3f9c005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9c0054e0 con 0x7f3fac0fde50 2024-09-13T15:29:37.124 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac19aac0 con 0x7f3fac0fde50 2024-09-13T15:29:37.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3f9c005640 con 0x7f3fac0fde50 2024-09-13T15:29:37.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3f9c03c990 con 0x7f3fac0fde50 2024-09-13T15:29:37.125 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.120+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fac19afe0 con 0x7f3fac0fde50 2024-09-13T15:29:37.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.121+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fac1085a0 con 0x7f3fac0fde50 2024-09-13T15:29:37.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.122+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 18) v1 ==== 90794+0+0 (secure 0 0 0) 0x7f3f9c0035f0 con 0x7f3fac0fde50 2024-09-13T15:29:37.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.122+0000 7f3fa9ffb640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3f9806cbd0 0x7f3f9806f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:37.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.122+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 3232+0+0 (secure 0 0 0) 0x7f3f9c0acad0 con 0x7f3fac0fde50 2024-09-13T15:29:37.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.122+0000 7f3fa3fff640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3f9806cbd0 0x7f3f9806f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:37.126 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.123+0000 7f3fa3fff640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3f9806cbd0 0x7f3f9806f090 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fac196970 tx=0x7f3f9400b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:29:37.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.127+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f3f9c077860 con 0x7f3fac0fde50 2024-09-13T15:29:37.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.301+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fac105780 con 0x7f3f9806cbd0 2024-09-13T15:29:37.305 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39300]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-13T15:29:37.305 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39300]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:37.305 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39300]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:37.305 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39300]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:37.326 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:37.324+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3f9c077c30 con 0x7f3fac0fde50 2024-09-13T15:29:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T15:29:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi090 2024-09-13T15:29:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T15:29:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:29:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:29:37.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:37.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:29:37.502 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:29:37.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39303]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39303]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39303]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 sudo[39303]: pam_unix(sudo:session): session closed for user root 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi090 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi090"}]: dispatch 2024-09-13T15:29:37.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mon metadata", "id": "smithi175"}]: dispatch 2024-09-13T15:29:38.596 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='client.14314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: mgrmap e19: smithi090.fyvvpd(active, since 2m), standbys: smithi175.otzykk 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi090 to 3946M 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.597 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.598 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:38.598 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:38.598 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.598 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='client.14314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi090:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: mgrmap e19: smithi090.fyvvpd(active, since 2m), standbys: smithi175.otzykk 2024-09-13T15:29:38.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi090 to 3946M 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:38.687 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:39.598 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:39 smithi090 ceph-mon[27811]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:39.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:39 smithi175 ceph-mon[36128]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:40.600 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:40 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1407309335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "930bd74f-868e-4916-b6aa-d1729d711c5b"}]: dispatch 2024-09-13T15:29:40.600 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:40 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1407309335' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "930bd74f-868e-4916-b6aa-d1729d711c5b"}]': finished 2024-09-13T15:29:40.600 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:40 smithi090 ceph-mon[27811]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T15:29:40.601 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:40.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:40 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1407309335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "930bd74f-868e-4916-b6aa-d1729d711c5b"}]: dispatch 2024-09-13T15:29:40.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:40 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1407309335' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "930bd74f-868e-4916-b6aa-d1729d711c5b"}]': finished 2024-09-13T15:29:40.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:40 smithi175 ceph-mon[36128]: osdmap e23: 4 total, 3 up, 4 in 2024-09-13T15:29:40.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:41.458 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:41 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3858934699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:29:41.458 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:41 smithi090 ceph-mon[27811]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:41.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:41 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3858934699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:29:41.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:41 smithi175 ceph-mon[36128]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:43.834 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:43 smithi090 ceph-mon[27811]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:43 smithi175 ceph-mon[36128]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:45.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:45.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:45.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:45 smithi090 ceph-mon[27811]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:45.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:45.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:45.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:45 smithi175 ceph-mon[36128]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:29:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:46.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T15:29:46.668 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:46.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-13T15:29:46.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:29:47.649 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:47 smithi090 ceph-mon[27811]: Deploying daemon osd.3 on smithi090 2024-09-13T15:29:47.650 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:47 smithi090 ceph-mon[27811]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:47.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:47 smithi175 ceph-mon[36128]: Deploying daemon osd.3 on smithi090 2024-09-13T15:29:47.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:47 smithi175 ceph-mon[36128]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:48.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:29:49.797 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:49 smithi090 ceph-mon[27811]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:49 smithi175 ceph-mon[36128]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:51 smithi090 ceph-mon[27811]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:52.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:51 smithi175 ceph-mon[36128]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:51 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:29:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:51 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:51 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:53.686 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:53 smithi090 ceph-mon[27811]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:53.717 INFO:teuthology.orchestra.run.smithi090.stdout:Created osd(s) 3 on host 'smithi090' 2024-09-13T15:29:53.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.716+0000 7f3fa9ffb640 1 -- 172.21.15.90:0/2008766830 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3fac105780 con 0x7f3f9806cbd0 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.718+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3f9806cbd0 msgr2=0x7f3f9806f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.718+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3f9806cbd0 0x7f3f9806f090 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3fac196970 tx=0x7f3f9400b390 comp rx=0 tx=0).stop 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.718+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 msgr2=0x7f3fac195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.718+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac195450 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c006010 tx=0x7f3f9c005e50 comp rx=0 tx=0).stop 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 shutdown_connections 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3f9806cbd0 0x7f3f9806f090 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3fac100e30 0x7f3fac195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:53.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 --2- 172.21.15.90:0/2008766830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3fac0fde50 0x7f3fac195450 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 >> 172.21.15.90:0/2008766830 conn(0x7f3fac0f99c0 msgr2=0x7f3fac104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:29:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 shutdown_connections 2024-09-13T15:29:53.721 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:29:53.719+0000 7f3fb266f640 1 -- 172.21.15.90:0/2008766830 wait complete. 2024-09-13T15:29:54.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:53 smithi175 ceph-mon[36128]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:54.665 DEBUG:teuthology.orchestra.run.smithi090:osd.3> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.3.service 2024-09-13T15:29:54.668 INFO:tasks.cephadm:Deploying osd.4 on smithi175 with /dev/vg_nvme/lv_4... 2024-09-13T15:29:54.668 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-13T15:29:54.876 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:29:54.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:54 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:54.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:54 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:54.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:54 smithi175 ceph-mon[36128]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T15:29:55.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:54 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:55.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:54 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:55.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:54 smithi090 ceph-mon[27811]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-13T15:29:56.001 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3[66054]: 2024-09-13T15:29:55.733+0000 7f9edf56d640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T15:29:56.002 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3[66054]: 2024-09-13T15:29:55.741+0000 7f9edb396640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:29:56.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-mon[27811]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:56.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-mon[27811]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T15:29:56.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-mon[27811]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T15:29:56.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:56.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:55 smithi090 ceph-mon[27811]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:29:56.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:55 smithi175 ceph-mon[36128]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:56.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:55 smithi175 ceph-mon[36128]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-13T15:29:56.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:55 smithi175 ceph-mon[36128]: osdmap e24: 4 total, 3 up, 4 in 2024-09-13T15:29:56.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:55 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:56.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:55 smithi175 ceph-mon[36128]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]: dispatch 2024-09-13T15:29:56.512 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-13T15:29:56.610 INFO:teuthology.orchestra.run.smithi175.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-13T15:29:56.625 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10+0 records in 2024-09-13T15:29:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:10+0 records out 2024-09-13T15:29:56.625 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132625 s, 791 MB/s 2024-09-13T15:29:56.625 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping successful for: 2024-09-13T15:29:56.739 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:56 smithi090 ceph-mon[27811]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:29:56.739 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:56 smithi090 ceph-mon[27811]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T15:29:56.739 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:56 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:56.739 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:56 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:57.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:56 smithi175 ceph-mon[36128]: from='osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi090", "root=default"]}]': finished 2024-09-13T15:29:57.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:56 smithi175 ceph-mon[36128]: osdmap e25: 4 total, 3 up, 4 in 2024-09-13T15:29:57.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:56 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:57.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:56 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:57.215 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_4 2024-09-13T15:29:57.417 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:29:57.740 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:29:57.740 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] boot 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T15:29:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-13T15:29:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: osd.3 [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] boot 2024-09-13T15:29:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: osdmap e26: 4 total, 4 up, 4 in 2024-09-13T15:29:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-13T15:29:58.988 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 -- 172.21.15.175:0/3867678483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540ffcf0 msgr2=0x7f6054100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:58.988 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 --2- 172.21.15.175:0/3867678483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540ffcf0 0x7f6054100170 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6050009880 tx=0x7f605002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:29:58.988 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 -- 172.21.15.175:0/3867678483 shutdown_connections 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 --2- 172.21.15.175:0/3867678483 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540ffcf0 0x7f6054100170 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 --2- 172.21.15.175:0/3867678483 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60540feaf0 0x7f60540feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 -- 172.21.15.175:0/3867678483 >> 172.21.15.175:0/3867678483 conn(0x7f60540fa280 msgr2=0x7f60540fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 -- 172.21.15.175:0/3867678483 shutdown_connections 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.984+0000 7f605c33b640 1 -- 172.21.15.175:0/3867678483 wait complete. 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 Processor -- start 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 -- start start 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 0x7f6054195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60540ffcf0 0x7f6054195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6054195ff0 con 0x7f60540ffcf0 2024-09-13T15:29:58.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6054196160 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 0x7f6054195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 0x7f6054195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:38956/0 (socket says 172.21.15.175:38956) 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 -- 172.21.15.175:0/2421627488 learned_addr learned my addr 172.21.15.175:0/2421627488 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 -- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60540ffcf0 msgr2=0x7f6054195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60540ffcf0 0x7f6054195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 -- 172.21.15.175:0/2421627488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050009560 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605a0d7640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 0x7f6054195570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f604800b3d0 tx=0x7f604800b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6048011800 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6048011df0 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605419ac40 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.985+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f604800f2e0 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.986+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605419b1c0 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.986+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60541075b0 con 0x7f60540feaf0 2024-09-13T15:29:58.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.987+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6048011960 con 0x7f60540feaf0 2024-09-13T15:29:58.991 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.988+0000 7f60477fe640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f604006cc50 0x7f604006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:29:58.991 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.988+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3664+0+0 (secure 0 0 0) 0x7f604808b870 con 0x7f60540feaf0 2024-09-13T15:29:58.991 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.988+0000 7f60598d6640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f604006cc50 0x7f604006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:29:58.991 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.989+0000 7f60598d6640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f604006cc50 0x7f604006f110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f605002a9c0 tx=0x7f6050000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:29:58.993 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:58.992+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f60480562c0 con 0x7f60540feaf0 2024-09-13T15:29:59.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:58 smithi090 ceph-mon[27811]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T15:29:59.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:58 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:59.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:58 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:59.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:58 smithi175 ceph-mon[36128]: osdmap e27: 4 total, 4 up, 4 in 2024-09-13T15:29:59.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:58 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:59.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:58 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:29:59.162 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:29:59.161+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f605419b460 con 0x7f604006cc50 2024-09-13T15:30:00.037 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:59 smithi090 ceph-mon[27811]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:00.037 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:30:00.038 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:30:00.038 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:29:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:00.135 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:59 smithi175 ceph-mon[36128]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:00.136 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:30:00.136 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:30:00.136 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:29:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:01.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:00 smithi090 ceph-mon[27811]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:30:01.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:00 smithi090 ceph-mon[27811]: overall HEALTH_OK 2024-09-13T15:30:01.155 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:00 smithi175 ceph-mon[36128]: from='client.24141 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:30:01.155 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:00 smithi175 ceph-mon[36128]: overall HEALTH_OK 2024-09-13T15:30:02.001 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:02.001 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/891707506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "488b1c67-90e5-4c01-b5f8-775fea197fb4"}]: dispatch 2024-09-13T15:30:02.001 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "488b1c67-90e5-4c01-b5f8-775fea197fb4"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "488b1c67-90e5-4c01-b5f8-775fea197fb4"}]': finished 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.002 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:02.003 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:02.003 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:02.003 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/891707506' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "488b1c67-90e5-4c01-b5f8-775fea197fb4"}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "488b1c67-90e5-4c01-b5f8-775fea197fb4"}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "488b1c67-90e5-4c01-b5f8-775fea197fb4"}]': finished 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: osdmap e28: 5 total, 4 up, 5 in 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:02.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:02.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:02.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:02.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:02.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:02.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:03.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:02 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi090 2024-09-13T15:30:03.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:02 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi090 to 2960M 2024-09-13T15:30:03.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:02 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2978108244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:30:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:02 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi090 2024-09-13T15:30:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:02 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi090 to 2960M 2024-09-13T15:30:03.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:02 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/2978108244' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:30:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:03 smithi175 ceph-mon[36128]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:03 smithi090 ceph-mon[27811]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:06.091 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:05 smithi175 ceph-mon[36128]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:05 smithi090 ceph-mon[27811]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:07.939 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:07 smithi175 ceph-mon[36128]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:07.939 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:07 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T15:30:07.939 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:07 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:07 smithi090 ceph-mon[27811]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:07 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-13T15:30:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:07 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:09.089 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:08 smithi175 ceph-mon[36128]: Deploying daemon osd.4 on smithi175 2024-09-13T15:30:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:08 smithi090 ceph-mon[27811]: Deploying daemon osd.4 on smithi175 2024-09-13T15:30:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:09 smithi175 ceph-mon[36128]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:10.112 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:09 smithi090 ceph-mon[27811]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:11.949 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:11 smithi175 ceph-mon[36128]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:11 smithi090 ceph-mon[27811]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:13.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:13.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:13.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:13.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:13.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:13.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:14.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:13 smithi175 ceph-mon[36128]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:13 smithi090 ceph-mon[27811]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:15.987 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.986+0000 7f60477fe640 1 -- 172.21.15.175:0/2421627488 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f605419b460 con 0x7f604006cc50 2024-09-13T15:30:15.988 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 4 on host 'smithi175' 2024-09-13T15:30:15.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f604006cc50 msgr2=0x7f604006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:15.989 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f604006cc50 0x7f604006f110 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f605002a9c0 tx=0x7f6050000bc0 comp rx=0 tx=0).stop 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 msgr2=0x7f6054195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 0x7f6054195570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f604800b3d0 tx=0x7f604800b890 comp rx=0 tx=0).stop 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 shutdown_connections 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f604006cc50 0x7f604006f110 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60540ffcf0 0x7f6054195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 --2- 172.21.15.175:0/2421627488 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60540feaf0 0x7f6054195570 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 >> 172.21.15.175:0/2421627488 conn(0x7f60540fa280 msgr2=0x7f6054067c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 shutdown_connections 2024-09-13T15:30:15.990 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:15.988+0000 7f605c33b640 1 -- 172.21.15.175:0/2421627488 wait complete. 2024-09-13T15:30:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:15 smithi090 ceph-mon[27811]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:15 smithi090 ceph-mon[27811]: from='osd.4 [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T15:30:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:15 smithi090 ceph-mon[27811]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T15:30:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:15 smithi175 ceph-mon[36128]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:15 smithi175 ceph-mon[36128]: from='osd.4 [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T15:30:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:15 smithi175 ceph-mon[36128]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-13T15:30:16.633 DEBUG:teuthology.orchestra.run.smithi175:osd.4> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.4.service 2024-09-13T15:30:16.636 INFO:tasks.cephadm:Deploying osd.5 on smithi175 with /dev/vg_nvme/lv_3... 2024-09-13T15:30:16.636 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='osd.4 [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:16.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:16.882 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:30:17.117 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:30:16.858+0000 7f132157c640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T15:30:17.117 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:30:16 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:30:16.863+0000 7f131d3a5640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: osdmap e29: 5 total, 4 up, 5 in 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='osd.4 [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:17.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: osd.4 [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] boot 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T15:30:18.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: osdmap e30: 5 total, 4 up, 5 in 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: osd.4 [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] boot 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: osdmap e31: 5 total, 5 up, 5 in 2024-09-13T15:30:18.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-13T15:30:18.891 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-13T15:30:18.988 INFO:teuthology.orchestra.run.smithi175.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-13T15:30:18.988 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:18 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:30:18.988 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:18 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:30:18.988 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:18.988 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:18 smithi175 ceph-mon[36128]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T15:30:19.003 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10+0 records in 2024-09-13T15:30:19.003 INFO:teuthology.orchestra.run.smithi175.stderr:10+0 records out 2024-09-13T15:30:19.003 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131255 s, 799 MB/s 2024-09-13T15:30:19.003 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping successful for: 2024-09-13T15:30:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:18 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:30:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:18 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:30:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:18 smithi090 ceph-mon[27811]: osdmap e32: 5 total, 5 up, 5 in 2024-09-13T15:30:19.845 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_3 2024-09-13T15:30:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:19 smithi090 ceph-mon[27811]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:19 smithi090 ceph-mon[27811]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T15:30:20.253 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:19 smithi175 ceph-mon[36128]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:20.253 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:19 smithi175 ceph-mon[36128]: osdmap e33: 5 total, 5 up, 5 in 2024-09-13T15:30:20.798 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:30:21.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:21 smithi175 ceph-mon[36128]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T15:30:21.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:21 smithi175 ceph-mon[36128]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:21.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:21 smithi090 ceph-mon[27811]: osdmap e34: 5 total, 5 up, 5 in 2024-09-13T15:30:21.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:21 smithi090 ceph-mon[27811]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:22.440 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: osdmap e35: 5 total, 5 up, 5 in 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:22.441 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:22 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: osdmap e35: 5 total, 5 up, 5 in 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:22.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:22.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:22.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:22.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:22.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:22.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:22 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.058+0000 7f451b1d6640 1 -- 172.21.15.175:0/1259841610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 msgr2=0x7f4514105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.058+0000 7f451b1d6640 1 --2- 172.21.15.175:0/1259841610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514105350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4500009880 tx=0x7f450002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- 172.21.15.175:0/1259841610 shutdown_connections 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 --2- 172.21.15.175:0/1259841610 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514105350 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 --2- 172.21.15.175:0/1259841610 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4514100630 0x7f4514102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- 172.21.15.175:0/1259841610 >> 172.21.15.175:0/1259841610 conn(0x7f45140fa250 msgr2=0x7f45140fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- 172.21.15.175:0/1259841610 shutdown_connections 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- 172.21.15.175:0/1259841610 wait complete. 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 Processor -- start 2024-09-13T15:30:23.070 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- start start 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4514100630 0x7f45141954e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514195a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4514195f60 con 0x7f4514100630 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f451b1d6640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45141960d0 con 0x7f4514102f60 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.059+0000 7f4513fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514195a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4513fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514195a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:46974/0 (socket says 172.21.15.175:46974) 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4513fff640 1 -- 172.21.15.175:0/273908861 learned_addr learned my addr 172.21.15.175:0/273908861 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4518f72640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4514100630 0x7f45141954e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4513fff640 1 -- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4514100630 msgr2=0x7f45141954e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4513fff640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4514100630 0x7f45141954e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4513fff640 1 -- 172.21.15.175:0/273908861 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4500009560 con 0x7f4514102f60 2024-09-13T15:30:23.071 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4513fff640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514195a20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4500004e90 tx=0x7f4500004c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4500005560 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4500003a80 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f450003c9c0 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451419ab00 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.060+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451419afa0 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.061+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45141085c0 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.062+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4500003bf0 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.062+0000 7f4511ffb640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f450406cc50 0x7f450406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.062+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4206+0+0 (secure 0 0 0) 0x7f45000ad010 con 0x7f4514102f60 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.063+0000 7f4518f72640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f450406cc50 0x7f450406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.063+0000 7f4518f72640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f450406cc50 0x7f450406f110 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f450800bbe0 tx=0x7f450800b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:30:23.072 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.067+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f45000778f0 con 0x7f4514102f60 2024-09-13T15:30:23.241 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:23.239+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f45141039c0 con 0x7f450406cc50 2024-09-13T15:30:23.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:23 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi175 2024-09-13T15:30:23.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:23 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi175 to 14912M 2024-09-13T15:30:23.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:23 smithi175 ceph-mon[36128]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:23.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:23 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:30:23.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:23 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:30:23.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:23 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:23.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:23 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi175 2024-09-13T15:30:23.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:23 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi175 to 14912M 2024-09-13T15:30:23.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:23 smithi090 ceph-mon[27811]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:23.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:30:23.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:30:23.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:23 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:24.609 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:24 smithi175 ceph-mon[36128]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:30:24.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:24 smithi090 ceph-mon[27811]: from='client.24159 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:30:25.652 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:25 smithi090 ceph-mon[27811]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-13T15:30:25.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:25 smithi175 ceph-mon[36128]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-13T15:30:26.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:26 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/2443921607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e49d9-f113-4f29-9905-ebc5884e3471"}]: dispatch 2024-09-13T15:30:26.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:26 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e49d9-f113-4f29-9905-ebc5884e3471"}]: dispatch 2024-09-13T15:30:26.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:26 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a6e49d9-f113-4f29-9905-ebc5884e3471"}]': finished 2024-09-13T15:30:26.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:26 smithi175 ceph-mon[36128]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T15:30:26.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:26.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:26 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/1917903777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:30:26.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:26 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2443921607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e49d9-f113-4f29-9905-ebc5884e3471"}]: dispatch 2024-09-13T15:30:26.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:26 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a6e49d9-f113-4f29-9905-ebc5884e3471"}]: dispatch 2024-09-13T15:30:26.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:26 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a6e49d9-f113-4f29-9905-ebc5884e3471"}]': finished 2024-09-13T15:30:26.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:26 smithi090 ceph-mon[27811]: osdmap e36: 6 total, 5 up, 6 in 2024-09-13T15:30:26.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:26.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:26 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/1917903777' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:30:27.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:27 smithi090 ceph-mon[27811]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-13T15:30:27.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:27 smithi175 ceph-mon[36128]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-13T15:30:29.761 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:29 smithi175 ceph-mon[36128]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T15:30:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:29 smithi090 ceph-mon[27811]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T15:30:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:31 smithi090 ceph-mon[27811]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-13T15:30:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:31 smithi175 ceph-mon[36128]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-13T15:30:32.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:32 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T15:30:32.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:32 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:32.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:32 smithi175 ceph-mon[36128]: Deploying daemon osd.5 on smithi175 2024-09-13T15:30:33.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:32 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-13T15:30:33.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:32 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:33.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:32 smithi090 ceph-mon[27811]: Deploying daemon osd.5 on smithi175 2024-09-13T15:30:33.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:33 smithi175 ceph-mon[36128]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:30:33.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:33 smithi090 ceph-mon[27811]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:30:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:35.946 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:35 smithi175 ceph-mon[36128]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:35 smithi090 ceph-mon[27811]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:37 smithi090 ceph-mon[27811]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:38.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:37 smithi175 ceph-mon[36128]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:38.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:40.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:39 smithi090 ceph-mon[27811]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:40.013 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:39 smithi175 ceph-mon[36128]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:40.065 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.063+0000 7f4511ffb640 1 -- 172.21.15.175:0/273908861 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f45141039c0 con 0x7f450406cc50 2024-09-13T15:30:40.065 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 5 on host 'smithi175' 2024-09-13T15:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f450406cc50 msgr2=0x7f450406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f450406cc50 0x7f450406f110 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f450800bbe0 tx=0x7f450800b390 comp rx=0 tx=0).stop 2024-09-13T15:30:40.067 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 msgr2=0x7f4514195a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514195a20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4500004e90 tx=0x7f4500004c60 comp rx=0 tx=0).stop 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 shutdown_connections 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f450406cc50 0x7f450406f110 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4514102f60 0x7f4514195a20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 --2- 172.21.15.175:0/273908861 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4514100630 0x7f45141954e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 >> 172.21.15.175:0/273908861 conn(0x7f45140fa250 msgr2=0x7f45140fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 shutdown_connections 2024-09-13T15:30:40.068 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:40.065+0000 7f451b1d6640 1 -- 172.21.15.175:0/273908861 wait complete. 2024-09-13T15:30:40.637 DEBUG:teuthology.orchestra.run.smithi175:osd.5> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.5.service 2024-09-13T15:30:40.639 INFO:tasks.cephadm:Deploying osd.6 on smithi175 with /dev/vg_nvme/lv_2... 2024-09-13T15:30:40.639 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-13T15:30:40.924 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:40 smithi175 ceph-mon[36128]: from='osd.5 [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T15:30:40.924 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:40 smithi175 ceph-mon[36128]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T15:30:40.924 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:40.924 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:41.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:40 smithi090 ceph-mon[27811]: from='osd.5 [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T15:30:41.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:40 smithi090 ceph-mon[27811]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-13T15:30:41.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:41.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:41.345 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:30:41.752 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:41 smithi175 ceph-mon[36128]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:41.752 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:41 smithi175 ceph-mon[36128]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T15:30:41.752 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:41 smithi175 ceph-mon[36128]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T15:30:41.752 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:41 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:41.753 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:41 smithi175 ceph-mon[36128]: from='osd.5 [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:41.753 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:41 smithi175 ceph-mon[36128]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:41 smithi090 ceph-mon[27811]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:41 smithi090 ceph-mon[27811]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-13T15:30:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:41 smithi090 ceph-mon[27811]: osdmap e37: 6 total, 5 up, 6 in 2024-09-13T15:30:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:41 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:41 smithi090 ceph-mon[27811]: from='osd.5 [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:42.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:41 smithi090 ceph-mon[27811]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:30:42.338 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:30:42.059+0000 7f8ee3871640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T15:30:42.338 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:30:42.066+0000 7f8edb692640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:30:42.863 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-mon[36128]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:30:42.863 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-mon[36128]: osdmap e38: 6 total, 5 up, 6 in 2024-09-13T15:30:42.863 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:42.863 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:42.863 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:42 smithi175 ceph-mon[36128]: from='osd.5 ' entity='osd.5' 2024-09-13T15:30:43.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:42 smithi090 ceph-mon[27811]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:30:43.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:42 smithi090 ceph-mon[27811]: osdmap e38: 6 total, 5 up, 6 in 2024-09-13T15:30:43.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:43.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:42 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:43.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:42 smithi090 ceph-mon[27811]: from='osd.5 ' entity='osd.5' 2024-09-13T15:30:43.110 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-13T15:30:43.199 INFO:teuthology.orchestra.run.smithi175.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-13T15:30:43.215 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10+0 records in 2024-09-13T15:30:43.215 INFO:teuthology.orchestra.run.smithi175.stderr:10+0 records out 2024-09-13T15:30:43.215 INFO:teuthology.orchestra.run.smithi175.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145773 s, 719 MB/s 2024-09-13T15:30:43.216 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping successful for: 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: osd.5 [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] boot 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:43.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:43 smithi175 ceph-mon[36128]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T15:30:43.860 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_2 2024-09-13T15:30:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:30:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:30:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-13T15:30:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: osd.5 [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] boot 2024-09-13T15:30:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: osdmap e39: 6 total, 6 up, 6 in 2024-09-13T15:30:44.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-13T15:30:44.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:44.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:44.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:43 smithi090 ceph-mon[27811]: osdmap e40: 6 total, 6 up, 6 in 2024-09-13T15:30:44.556 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:30:45.875 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:45 smithi175 ceph-mon[36128]: pgmap v131: 1 pgs: 1 remapped; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:30:45.875 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:45 smithi175 ceph-mon[36128]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T15:30:45.875 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:45 smithi175 ceph-mon[36128]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T15:30:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:45 smithi090 ceph-mon[27811]: pgmap v131: 1 pgs: 1 remapped; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:30:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:45 smithi090 ceph-mon[27811]: osdmap e41: 6 total, 6 up, 6 in 2024-09-13T15:30:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:45 smithi090 ceph-mon[27811]: osdmap e42: 6 total, 6 up, 6 in 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.129+0000 7f18648f5640 1 -- 172.21.15.175:0/2758613 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600ffcf0 msgr2=0x7f1860100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.129+0000 7f18648f5640 1 --2- 172.21.15.175:0/2758613 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600ffcf0 0x7f1860100170 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1854009880 tx=0x7f185402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.130+0000 7f18648f5640 1 -- 172.21.15.175:0/2758613 shutdown_connections 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.130+0000 7f18648f5640 1 --2- 172.21.15.175:0/2758613 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600ffcf0 0x7f1860100170 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.130+0000 7f18648f5640 1 --2- 172.21.15.175:0/2758613 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18600feaf0 0x7f18600feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.130+0000 7f18648f5640 1 -- 172.21.15.175:0/2758613 >> 172.21.15.175:0/2758613 conn(0x7f18600fa280 msgr2=0x7f18600fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.130+0000 7f18648f5640 1 -- 172.21.15.175:0/2758613 shutdown_connections 2024-09-13T15:30:47.134 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.130+0000 7f18648f5640 1 -- 172.21.15.175:0/2758613 wait complete. 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f18648f5640 1 Processor -- start 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f18648f5640 1 -- start start 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f18648f5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 0x7f1860193380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f18648f5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18600ffcf0 0x7f18601938c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f18648f5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860193e00 con 0x7f18600ffcf0 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f18648f5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860193f70 con 0x7f18600feaf0 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 0x7f1860193380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 0x7f1860193380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:40890/0 (socket says 172.21.15.175:40890) 2024-09-13T15:30:47.135 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 -- 172.21.15.175:0/233266524 learned_addr learned my addr 172.21.15.175:0/233266524 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185dd9b640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18600ffcf0 0x7f18601938c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 -- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18600ffcf0 msgr2=0x7f18601938c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18600ffcf0 0x7f18601938c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 -- 172.21.15.175:0/233266524 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1854009560 con 0x7f18600feaf0 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.131+0000 7f185e59c640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 0x7f1860193380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f185000b3d0 tx=0x7f185000b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.132+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1850011800 con 0x7f18600feaf0 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.132+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1850011df0 con 0x7f18600feaf0 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.132+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1860198a50 con 0x7f18600feaf0 2024-09-13T15:30:47.136 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.132+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f185000f2e0 con 0x7f18600feaf0 2024-09-13T15:30:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.132+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1860198fd0 con 0x7f18600feaf0 2024-09-13T15:30:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.133+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18601075b0 con 0x7f18600feaf0 2024-09-13T15:30:47.137 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.133+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1850011960 con 0x7f18600feaf0 2024-09-13T15:30:47.138 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.134+0000 7f184f7fe640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f184806cc50 0x7f184806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:30:47.138 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.134+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4638+0+0 (secure 0 0 0) 0x7f185008bcc0 con 0x7f18600feaf0 2024-09-13T15:30:47.138 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.134+0000 7f185dd9b640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f184806cc50 0x7f184806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:30:47.138 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.135+0000 7f185dd9b640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f184806cc50 0x7f184806f110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18601948a0 tx=0x7f1854004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:30:47.140 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.138+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1850056340 con 0x7f18600feaf0 2024-09-13T15:30:47.315 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:47 smithi175 ceph-mon[36128]: osdmap e43: 6 total, 6 up, 6 in 2024-09-13T15:30:47.315 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:47 smithi175 ceph-mon[36128]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:30:47.316 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:30:47.313+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1860199800 con 0x7f184806cc50 2024-09-13T15:30:47.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:47 smithi090 ceph-mon[27811]: osdmap e43: 6 total, 6 up, 6 in 2024-09-13T15:30:47.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:47 smithi090 ceph-mon[27811]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 562 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi175 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:48.581 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi175 to 7456M 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:48.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='client.24177 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:30:48.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:30:48.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:30:48.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi175 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi175 to 7456M 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:30:48.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:30:49.410 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:49 smithi175 ceph-mon[36128]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T15:30:49.410 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:49.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:49 smithi090 ceph-mon[27811]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T15:30:49.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:30:50.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:50 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/3100024449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb"}]: dispatch 2024-09-13T15:30:50.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:50 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb"}]: dispatch 2024-09-13T15:30:50.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:50 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb"}]': finished 2024-09-13T15:30:50.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:50 smithi090 ceph-mon[27811]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T15:30:50.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:50 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:30:50.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:50 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/84236080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:30:50.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:50 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/3100024449' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb"}]: dispatch 2024-09-13T15:30:50.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:50 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb"}]: dispatch 2024-09-13T15:30:50.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:50 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb"}]': finished 2024-09-13T15:30:50.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:50 smithi175 ceph-mon[36128]: osdmap e44: 7 total, 6 up, 7 in 2024-09-13T15:30:50.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:50 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:30:50.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:50 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/84236080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:30:51.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:51 smithi090 ceph-mon[27811]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T15:30:51.769 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:51 smithi175 ceph-mon[36128]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-13T15:30:53.882 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:53 smithi175 ceph-mon[36128]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T15:30:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:53 smithi090 ceph-mon[27811]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-13T15:30:55.860 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:55 smithi175 ceph-mon[36128]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T15:30:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:55 smithi090 ceph-mon[27811]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T15:30:56.941 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:56 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T15:30:56.941 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:56 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:56.942 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:56 smithi175 ceph-mon[36128]: Deploying daemon osd.6 on smithi175 2024-09-13T15:30:57.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:56 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-13T15:30:57.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:56 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:30:57.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:56 smithi090 ceph-mon[27811]: Deploying daemon osd.6 on smithi175 2024-09-13T15:30:57.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:57 smithi090 ceph-mon[27811]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:30:57.853 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:57 smithi175 ceph-mon[36128]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:30:59.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:30:59 smithi175 ceph-mon[36128]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:30:59 smithi090 ceph-mon[27811]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:01.633 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:01 smithi175 ceph-mon[36128]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:01.633 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:01.633 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:01.633 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:01 smithi090 ceph-mon[27811]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:03.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:03 smithi175 ceph-mon[36128]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:03.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:03 smithi090 ceph-mon[27811]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:04.714 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 6 on host 'smithi175' 2024-09-13T15:31:04.714 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.712+0000 7f184f7fe640 1 -- 172.21.15.175:0/233266524 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1860199800 con 0x7f184806cc50 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.714+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f184806cc50 msgr2=0x7f184806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.714+0000 7f18648f5640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f184806cc50 0x7f184806f110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f18601948a0 tx=0x7f1854004f00 comp rx=0 tx=0).stop 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.714+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 msgr2=0x7f1860193380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.714+0000 7f18648f5640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 0x7f1860193380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f185000b3d0 tx=0x7f185000b890 comp rx=0 tx=0).stop 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.714+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 shutdown_connections 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.715+0000 7f18648f5640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f184806cc50 0x7f184806f110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.715+0000 7f18648f5640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f18600ffcf0 0x7f18601938c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.715+0000 7f18648f5640 1 --2- 172.21.15.175:0/233266524 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18600feaf0 0x7f1860193380 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.715+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 >> 172.21.15.175:0/233266524 conn(0x7f18600fa280 msgr2=0x7f1860067c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.715+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 shutdown_connections 2024-09-13T15:31:04.716 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:04.715+0000 7f18648f5640 1 -- 172.21.15.175:0/233266524 wait complete. 2024-09-13T15:31:05.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:04 smithi090 ceph-mon[27811]: from='osd.6 [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T15:31:05.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:04 smithi090 ceph-mon[27811]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T15:31:05.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:04 smithi175 ceph-mon[36128]: from='osd.6 [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T15:31:05.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:04 smithi175 ceph-mon[36128]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-13T15:31:05.316 DEBUG:teuthology.orchestra.run.smithi175:osd.6> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.6.service 2024-09-13T15:31:05.319 INFO:tasks.cephadm:Deploying osd.7 on smithi175 with /dev/vg_nvme/lv_1... 2024-09-13T15:31:05.319 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-13T15:31:05.744 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: from='osd.6 [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:05.872 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:05 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-13T15:31:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: osdmap e45: 7 total, 6 up, 7 in 2024-09-13T15:31:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: from='osd.6 [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:06.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:06.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:06.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:05 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:06.190 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:31:06.043+0000 7f4919a43640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T15:31:06.190 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:31:06.050+0000 7f4911864640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:31:06.810 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-mon[36128]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:31:06.810 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-mon[36128]: osdmap e46: 7 total, 6 up, 7 in 2024-09-13T15:31:06.810 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:06.810 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:06.810 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:06 smithi175 ceph-mon[36128]: from='osd.6 ' entity='osd.6' 2024-09-13T15:31:07.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:06 smithi090 ceph-mon[27811]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:31:07.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:06 smithi090 ceph-mon[27811]: osdmap e46: 7 total, 6 up, 7 in 2024-09-13T15:31:07.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:07.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:07.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:06 smithi090 ceph-mon[27811]: from='osd.6 ' entity='osd.6' 2024-09-13T15:31:07.495 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-13T15:31:07.599 INFO:teuthology.orchestra.run.smithi175.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-13T15:31:07.616 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10+0 records in 2024-09-13T15:31:07.616 INFO:teuthology.orchestra.run.smithi175.stderr:10+0 records out 2024-09-13T15:31:07.616 INFO:teuthology.orchestra.run.smithi175.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138772 s, 756 MB/s 2024-09-13T15:31:07.616 INFO:teuthology.orchestra.run.smithi175.stderr:--> Zapping successful for: 2024-09-13T15:31:07.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: osd.6 [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] boot 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:07.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:07 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:31:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:31:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-13T15:31:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: osd.6 [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] boot 2024-09-13T15:31:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: osdmap e47: 7 total, 7 up, 7 in 2024-09-13T15:31:08.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-13T15:31:08.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:08.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:07 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:08.929 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:08 smithi175 ceph-mon[36128]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T15:31:09.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:08 smithi090 ceph-mon[27811]: osdmap e48: 7 total, 7 up, 7 in 2024-09-13T15:31:09.344 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi175:vg_nvme/lv_1 2024-09-13T15:31:09.901 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:31:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:09 smithi090 ceph-mon[27811]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:09 smithi090 ceph-mon[27811]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T15:31:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:09 smithi175 ceph-mon[36128]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:09 smithi175 ceph-mon[36128]: osdmap e49: 7 total, 7 up, 7 in 2024-09-13T15:31:10.959 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:10 smithi175 ceph-mon[36128]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T15:31:11.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:10 smithi090 ceph-mon[27811]: osdmap e50: 7 total, 7 up, 7 in 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: osdmap e51: 7 total, 7 up, 7 in 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:11.843 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:11.844 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:11.844 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: pgmap v154: 1 pgs: 1 remapped+peering; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: osdmap e51: 7 total, 7 up, 7 in 2024-09-13T15:31:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:12.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:12.099 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 -- 172.21.15.175:0/1412998833 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 msgr2=0x7f1fe4100120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 --2- 172.21.15.175:0/1412998833 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe4100120 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4009880 tx=0x7f1fd402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 -- 172.21.15.175:0/1412998833 shutdown_connections 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 --2- 172.21.15.175:0/1412998833 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe4100120 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 --2- 172.21.15.175:0/1412998833 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe40feaa0 0x7f1fe40feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 -- 172.21.15.175:0/1412998833 >> 172.21.15.175:0/1412998833 conn(0x7f1fe40fa250 msgr2=0x7f1fe40fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 -- 172.21.15.175:0/1412998833 shutdown_connections 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.096+0000 7f1fec381640 1 -- 172.21.15.175:0/1412998833 wait complete. 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fec381640 1 Processor -- start 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fec381640 1 -- start start 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fec381640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe40feaa0 0x7f1fe41954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fec381640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe41959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:12.100 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fec381640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe4195f30 con 0x7f1fe40feaa0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fec381640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe41960a0 con 0x7f1fe40ffca0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fe991c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe41959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fe991c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe41959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59908/0 (socket says 172.21.15.175:59908) 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fe991c640 1 -- 172.21.15.175:0/4283496279 learned_addr learned my addr 172.21.15.175:0/4283496279 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fea11d640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe40feaa0 0x7f1fe41954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fe991c640 1 -- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe40feaa0 msgr2=0x7f1fe41954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fe991c640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe40feaa0 0x7f1fe41954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.097+0000 7f1fe991c640 1 -- 172.21.15.175:0/4283496279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4009560 con 0x7f1fe40ffca0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.098+0000 7f1fe991c640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe41959f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4005940 tx=0x7f1fd4005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.098+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd4005630 con 0x7f1fe40ffca0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.098+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1fd4003a80 con 0x7f1fe40ffca0 2024-09-13T15:31:12.101 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.098+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe419ab20 con 0x7f1fe40ffca0 2024-09-13T15:31:12.102 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.098+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1fd403cb20 con 0x7f1fe40ffca0 2024-09-13T15:31:12.102 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.098+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe419b010 con 0x7f1fe40ffca0 2024-09-13T15:31:12.103 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.099+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe41074e0 con 0x7f1fe40ffca0 2024-09-13T15:31:12.103 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.099+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1fd4003bf0 con 0x7f1fe40ffca0 2024-09-13T15:31:12.103 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.100+0000 7f1fdb7fe640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1fd006cc50 0x7f1fd006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:12.103 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.100+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5070+0+0 (secure 0 0 0) 0x7f1fd40ad400 con 0x7f1fe40ffca0 2024-09-13T15:31:12.103 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.100+0000 7f1fea11d640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1fd006cc50 0x7f1fd006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:12.103 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.101+0000 7f1fea11d640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1fd006cc50 0x7f1fd006f110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00058c0 tx=0x7f1fe000a2d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:12.105 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.104+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1fd4077a00 con 0x7f1fe40ffca0 2024-09-13T15:31:12.277 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:12.276+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fe40fc480 con 0x7f1fd006cc50 2024-09-13T15:31:12.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi175 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi175 to 4970M 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:31:12.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:12 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:13.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi175 2024-09-13T15:31:13.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi175 to 4970M 2024-09-13T15:31:13.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:13.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:13.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:13.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:13.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-13T15:31:13.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-13T15:31:13.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:12 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:13.880 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:13 smithi175 ceph-mon[36128]: from='client.24195 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:31:13.880 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:13 smithi175 ceph-mon[36128]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T15:31:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:13 smithi090 ceph-mon[27811]: from='client.24195 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi175:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:31:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:13 smithi090 ceph-mon[27811]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-09-13T15:31:15.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:14 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/3936311838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "325ddbd7-1fe4-494f-a513-865f12961c78"}]: dispatch 2024-09-13T15:31:15.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:14 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "325ddbd7-1fe4-494f-a513-865f12961c78"}]: dispatch 2024-09-13T15:31:15.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:14 smithi090 ceph-mon[27811]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "325ddbd7-1fe4-494f-a513-865f12961c78"}]': finished 2024-09-13T15:31:15.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:14 smithi090 ceph-mon[27811]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T15:31:15.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:15.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:14 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/3936311838' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "325ddbd7-1fe4-494f-a513-865f12961c78"}]: dispatch 2024-09-13T15:31:15.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:14 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "325ddbd7-1fe4-494f-a513-865f12961c78"}]: dispatch 2024-09-13T15:31:15.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:14 smithi175 ceph-mon[36128]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "325ddbd7-1fe4-494f-a513-865f12961c78"}]': finished 2024-09-13T15:31:15.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:14 smithi175 ceph-mon[36128]: osdmap e52: 8 total, 7 up, 8 in 2024-09-13T15:31:15.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:15 smithi090 ceph-mon[27811]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T15:31:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/2227213668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:31:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:15 smithi175 ceph-mon[36128]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T15:31:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:15 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/2227213668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-13T15:31:18.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:17 smithi090 ceph-mon[27811]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T15:31:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:17 smithi175 ceph-mon[36128]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-09-13T15:31:19.004 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:20.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:19 smithi175 ceph-mon[36128]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T15:31:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:19 smithi090 ceph-mon[27811]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-13T15:31:20.932 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:20 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T15:31:20.932 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:20 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:20 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-13T15:31:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:20 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:22.025 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:21 smithi175 ceph-mon[36128]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:31:22.026 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:21 smithi175 ceph-mon[36128]: Deploying daemon osd.7 on smithi175 2024-09-13T15:31:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:21 smithi090 ceph-mon[27811]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:31:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:21 smithi090 ceph-mon[27811]: Deploying daemon osd.7 on smithi175 2024-09-13T15:31:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:23 smithi175 ceph-mon[36128]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:23 smithi090 ceph-mon[27811]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:25.914 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:25 smithi175 ceph-mon[36128]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:25.914 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:25.914 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:25.914 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:26.053 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:25 smithi090 ceph-mon[27811]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:26.053 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:26.053 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:26.054 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:28.156 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:27 smithi175 ceph-mon[36128]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:27 smithi090 ceph-mon[27811]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:28.342 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.340+0000 7f1fdb7fe640 1 -- 172.21.15.175:0/4283496279 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1fe40fc480 con 0x7f1fd006cc50 2024-09-13T15:31:28.342 INFO:teuthology.orchestra.run.smithi175.stdout:Created osd(s) 7 on host 'smithi175' 2024-09-13T15:31:28.344 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1fd006cc50 msgr2=0x7f1fd006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1fd006cc50 0x7f1fd006f110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00058c0 tx=0x7f1fe000a2d0 comp rx=0 tx=0).stop 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 msgr2=0x7f1fe41959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe41959f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1fd4005940 tx=0x7f1fd4005230 comp rx=0 tx=0).stop 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 shutdown_connections 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f1fd006cc50 0x7f1fd006f110 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1fe40ffca0 0x7f1fe41959f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 --2- 172.21.15.175:0/4283496279 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1fe40feaa0 0x7f1fe41954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 >> 172.21.15.175:0/4283496279 conn(0x7f1fe40fa250 msgr2=0x7f1fe40fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 shutdown_connections 2024-09-13T15:31:28.345 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:28.343+0000 7f1fec381640 1 -- 172.21.15.175:0/4283496279 wait complete. 2024-09-13T15:31:29.047 DEBUG:teuthology.orchestra.run.smithi175:osd.7> sudo journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.7.service 2024-09-13T15:31:29.050 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-13T15:31:29.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T15:31:29.265 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:29.605 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:29.606 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:29 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:29.606 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:29 smithi090 ceph-mon[27811]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:29.606 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:29 smithi090 ceph-mon[27811]: from='osd.7 [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T15:31:29.606 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:29 smithi090 ceph-mon[27811]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T15:31:29.635 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:29.635 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:29 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:29.635 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:29 smithi175 ceph-mon[36128]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:29.635 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:29 smithi175 ceph-mon[36128]: from='osd.7 [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T15:31:29.635 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:29 smithi175 ceph-mon[36128]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-13T15:31:30.701 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:30 smithi090 ceph-mon[27811]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T15:31:30.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:30 smithi090 ceph-mon[27811]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T15:31:30.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:30 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:30.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:30 smithi090 ceph-mon[27811]: from='osd.7 [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:30.702 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:30 smithi090 ceph-mon[27811]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:30.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-mon[36128]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-13T15:31:30.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-mon[36128]: osdmap e53: 8 total, 7 up, 8 in 2024-09-13T15:31:30.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:30.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-mon[36128]: from='osd.7 [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:30.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-mon[36128]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]: dispatch 2024-09-13T15:31:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.894+0000 7fe03424d640 1 -- 172.21.15.90:0/1893408839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c0744e0 msgr2=0x7fe02c072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.894+0000 7fe03424d640 1 --2- 172.21.15.90:0/1893408839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c0744e0 0x7fe02c072940 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe028009880 tx=0x7fe02802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:31:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.894+0000 7fe03424d640 1 -- 172.21.15.90:0/1893408839 shutdown_connections 2024-09-13T15:31:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.894+0000 7fe03424d640 1 --2- 172.21.15.90:0/1893408839 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe02c072f10 0x7fe02c073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.894+0000 7fe03424d640 1 --2- 172.21.15.90:0/1893408839 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c0744e0 0x7fe02c072940 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:30.924 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.894+0000 7fe03424d640 1 -- 172.21.15.90:0/1893408839 >> 172.21.15.90:0/1893408839 conn(0x7fe02c0fb330 msgr2=0x7fe02c0fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 -- 172.21.15.90:0/1893408839 shutdown_connections 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 -- 172.21.15.90:0/1893408839 wait complete. 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 Processor -- start 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 -- start start 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 0x7fe02c19ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe02c0744e0 0x7fe02c19bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c19c4c0 con 0x7fe02c072f10 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe03424d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c19c630 con 0x7fe02c0744e0 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe031fe9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 0x7fe02c19ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe031fe9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 0x7fe02c19ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55458/0 (socket says 172.21.15.90:55458) 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.895+0000 7fe031fe9640 1 -- 172.21.15.90:0/2756221343 learned_addr learned my addr 172.21.15.90:0/2756221343 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:30.925 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe0317e8640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe02c0744e0 0x7fe02c19bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe031fe9640 1 -- 172.21.15.90:0/2756221343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe02c0744e0 msgr2=0x7fe02c19bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe031fe9640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe02c0744e0 0x7fe02c19bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe031fe9640 1 -- 172.21.15.90:0/2756221343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009560 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe031fe9640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 0x7fe02c19ba40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe028006010 tx=0x7fe028005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe028005600 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe028003a80 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c1a10b0 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe02803ca90 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.896+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c1a15d0 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.897+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c10eb90 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.898+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe028003bf0 con 0x7fe02c072f10 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.898+0000 7fe022ffd640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe01806cc50 0x7fe01806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:30.926 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.898+0000 7fe0317e8640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe01806cc50 0x7fe01806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.898+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5222+0+0 (secure 0 0 0) 0x7fe0280ad510 con 0x7fe02c072f10 2024-09-13T15:31:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.898+0000 7fe0317e8640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe01806cc50 0x7fe01806f110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe01c005e10 tx=0x7fe01c005da0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:30.927 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:30.902+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fe028077940 con 0x7fe02c072f10 2024-09-13T15:31:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.070+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe02c1a19d0 con 0x7fe02c072f10 2024-09-13T15:31:31.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.070+0000 7fe022ffd640 1 -- 172.21.15.90:0/2756221343 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v54) v1 ==== 74+0+130 (secure 0 0 0) 0x7fe02807af60 con 0x7fe02c072f10 2024-09-13T15:31:31.073 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:31:31.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe01806cc50 msgr2=0x7fe01806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe01806cc50 0x7fe01806f110 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe01c005e10 tx=0x7fe01c005da0 comp rx=0 tx=0).stop 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 msgr2=0x7fe02c19ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 0x7fe02c19ba40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe028006010 tx=0x7fe028005000 comp rx=0 tx=0).stop 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 shutdown_connections 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe01806cc50 0x7fe01806f110 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe02c0744e0 0x7fe02c19bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 --2- 172.21.15.90:0/2756221343 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe02c072f10 0x7fe02c19ba40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 >> 172.21.15.90:0/2756221343 conn(0x7fe02c0fb330 msgr2=0x7fe02c101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 shutdown_connections 2024-09-13T15:31:31.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:31.072+0000 7fe03424d640 1 -- 172.21.15.90:0/2756221343 wait complete. 2024-09-13T15:31:31.088 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:31:30.791+0000 7f814eebb640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T15:31:31.088 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:31:30 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:31:30.798+0000 7f8146cdc640 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: osdmap e54: 8 total, 7 up, 8 in 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:31.642 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='osd.7 ' entity='osd.7' 2024-09-13T15:31:31.643 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2756221343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T15:31:31.643 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: osd.7 [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] boot 2024-09-13T15:31:31.643 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T15:31:31.643 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:31 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:31.730 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi175", "root=default"]}]': finished 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: osdmap e54: 8 total, 7 up, 8 in 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='osd.7 ' entity='osd.7' 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2756221343' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: osd.7 [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] boot 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: osdmap e55: 8 total, 8 up, 8 in 2024-09-13T15:31:31.731 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:31 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-13T15:31:31.754 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1726241466,"num_in_osds":8,"osd_in_since":1726241474,"num_remapped_pgs":0} 2024-09-13T15:31:32.627 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:32 smithi175 ceph-mon[36128]: purged_snaps scrub starts 2024-09-13T15:31:32.627 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:32 smithi175 ceph-mon[36128]: purged_snaps scrub ok 2024-09-13T15:31:32.628 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:32 smithi175 ceph-mon[36128]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T15:31:32.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:32 smithi090 ceph-mon[27811]: purged_snaps scrub starts 2024-09-13T15:31:32.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:32 smithi090 ceph-mon[27811]: purged_snaps scrub ok 2024-09-13T15:31:32.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:32 smithi090 ceph-mon[27811]: osdmap e56: 8 total, 8 up, 8 in 2024-09-13T15:31:32.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-13T15:31:32.967 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:33.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:33.582 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:33.583 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:33.584 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:33.752 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: osdmap e57: 8 total, 8 up, 8 in 2024-09-13T15:31:33.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:31:33.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:31:33.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:31:33.753 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 -- 172.21.15.90:0/2303111337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248100610 msgr2=0x7fa248102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 --2- 172.21.15.90:0/2303111337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248100610 0x7fa248102a00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa2380098b0 tx=0x7fa23802a4d0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 -- 172.21.15.90:0/2303111337 shutdown_connections 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 --2- 172.21.15.90:0/2303111337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa248102f40 0x7fa248105330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 --2- 172.21.15.90:0/2303111337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248100610 0x7fa248102a00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 -- 172.21.15.90:0/2303111337 >> 172.21.15.90:0/2303111337 conn(0x7fa2480fa230 msgr2=0x7fa2480fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 -- 172.21.15.90:0/2303111337 shutdown_connections 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.681+0000 7fa24c85a640 1 -- 172.21.15.90:0/2303111337 wait complete. 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa24c85a640 1 Processor -- start 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa24c85a640 1 -- start start 2024-09-13T15:31:34.688 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa24c85a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa248100610 0x7fa2481955a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa24c85a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 0x7fa248195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa24c85a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa248196020 con 0x7fa248102f40 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa24c85a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa248196190 con 0x7fa248100610 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa245d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 0x7fa248195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa245d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 0x7fa248195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47828/0 (socket says 172.21.15.90:47828) 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.682+0000 7fa245d9b640 1 -- 172.21.15.90:0/3388949936 learned_addr learned my addr 172.21.15.90:0/3388949936 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa245d9b640 1 -- 172.21.15.90:0/3388949936 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa248100610 msgr2=0x7fa2481955a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa24659c640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa248100610 0x7fa2481955a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa245d9b640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa248100610 0x7fa2481955a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa245d9b640 1 -- 172.21.15.90:0/3388949936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa238009560 con 0x7fa248102f40 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa245d9b640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 0x7fa248195ae0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00b3d0 tx=0x7fa23c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa23c011800 con 0x7fa248102f40 2024-09-13T15:31:34.689 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa23c011df0 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa23c010ea0 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa24819ac70 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.683+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa24819b1f0 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.684+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa248108640 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.684+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa23c011960 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.685+0000 7fa2377fe640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa23006cc50 0x7fa23006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.685+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fa23c08bf40 con 0x7fa248102f40 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.685+0000 7fa24659c640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa23006cc50 0x7fa23006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:34.690 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.685+0000 7fa24659c640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa23006cc50 0x7fa23006f110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa238005ec0 tx=0x7fa238000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:34.691 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.689+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa23c0bfe20 con 0x7fa248102f40 2024-09-13T15:31:34.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:34 smithi090 ceph-mon[27811]: Detected new or changed devices on smithi175 2024-09-13T15:31:34.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:34 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi175 to 3728M 2024-09-13T15:31:34.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:34.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:34 smithi090 ceph-mon[27811]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T15:31:34.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:34 smithi175 ceph-mon[36128]: Detected new or changed devices on smithi175 2024-09-13T15:31:34.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:34 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi175 to 3728M 2024-09-13T15:31:34.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:34.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:34 smithi175 ceph-mon[36128]: osdmap e58: 8 total, 8 up, 8 in 2024-09-13T15:31:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.856+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa248108820 con 0x7fa248102f40 2024-09-13T15:31:34.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.856+0000 7fa2377fe640 1 -- 172.21.15.90:0/3388949936 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v58) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa23c0150a0 con 0x7fa248102f40 2024-09-13T15:31:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:31:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa23006cc50 msgr2=0x7fa23006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:34.859 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa23006cc50 0x7fa23006f110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa238005ec0 tx=0x7fa238000bc0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 msgr2=0x7fa248195ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 0x7fa248195ae0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa23c00b3d0 tx=0x7fa23c00b890 comp rx=0 tx=0).stop 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 shutdown_connections 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa23006cc50 0x7fa23006f110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa248102f40 0x7fa248195ae0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 --2- 172.21.15.90:0/3388949936 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa248100610 0x7fa2481955a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 >> 172.21.15.90:0/3388949936 conn(0x7fa2480fa230 msgr2=0x7fa2480fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 shutdown_connections 2024-09-13T15:31:34.860 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:34.858+0000 7fa24c85a640 1 -- 172.21.15.90:0/3388949936 wait complete. 2024-09-13T15:31:35.598 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1726241491,"num_in_osds":8,"osd_in_since":1726241474,"num_remapped_pgs":0} 2024-09-13T15:31:35.599 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:31:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:35 smithi090 ceph-mon[27811]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:35 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3388949936' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T15:31:35.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:35 smithi090 ceph-mon[27811]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T15:31:35.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:35 smithi175 ceph-mon[36128]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 617 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:35.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:35 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3388949936' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-13T15:31:35.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:35 smithi175 ceph-mon[36128]: osdmap e59: 8 total, 8 up, 8 in 2024-09-13T15:31:35.799 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:37.307 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 -- 172.21.15.90:0/873241996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 msgr2=0x7fc6cc0fee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 --2- 172.21.15.90:0/873241996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc0fee90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc009c20 tx=0x7fc6bc02a970 comp rx=0 tx=0).stop 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 -- 172.21.15.90:0/873241996 shutdown_connections 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 --2- 172.21.15.90:0/873241996 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc0fee90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 --2- 172.21.15.90:0/873241996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc6cc104a30 0x7fc6cc104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 -- 172.21.15.90:0/873241996 >> 172.21.15.90:0/873241996 conn(0x7fc6cc0fa730 msgr2=0x7fc6cc0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 -- 172.21.15.90:0/873241996 shutdown_connections 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.302+0000 7fc6d3113640 1 -- 172.21.15.90:0/873241996 wait complete. 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d3113640 1 Processor -- start 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d3113640 1 -- start start 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d3113640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc19b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:37.308 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d3113640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc6cc104a30 0x7fc6cc19bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d3113640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc19c210 con 0x7fc6cc0fea30 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d3113640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc1957e0 con 0x7fc6cc104a30 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d0eaf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc19b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d0eaf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc19b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47852/0 (socket says 172.21.15.90:47852) 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d0eaf640 1 -- 172.21.15.90:0/923751637 learned_addr learned my addr 172.21.15.90:0/923751637 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6cbfff640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc6cc104a30 0x7fc6cc19bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d0eaf640 1 -- 172.21.15.90:0/923751637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc6cc104a30 msgr2=0x7fc6cc19bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.303+0000 7fc6d0eaf640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc6cc104a30 0x7fc6cc19bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6d0eaf640 1 -- 172.21.15.90:0/923751637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc009900 con 0x7fc6cc0fea30 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6d0eaf640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc19b6d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400d6f0 tx=0x7fc6c400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4009a30 con 0x7fc6cc0fea30 2024-09-13T15:31:37.309 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6c40054b0 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc195a60 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6c4004d00 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.304+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc195f80 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.305+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc100170 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.305+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc6c4005020 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.306+0000 7fc6c9ffb640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc6b806cc50 0x7fc6b806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.306+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fc6c408bdd0 con 0x7fc6cc0fea30 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.306+0000 7fc6cbfff640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc6b806cc50 0x7fc6b806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:37.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.306+0000 7fc6cbfff640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc6b806cc50 0x7fc6b806f110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc196b30 tx=0x7fc6bc03d040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:37.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.310+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fc6c40561a0 con 0x7fc6cc0fea30 2024-09-13T15:31:37.472 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.471+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6cc196d10 con 0x7fc6cc0fea30 2024-09-13T15:31:37.473 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.472+0000 7fc6c9ffb640 1 -- 172.21.15.90:0/923751637 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13848 (secure 0 0 0) 0x7fc6c4020090 con 0x7fc6cc0fea30 2024-09-13T15:31:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:31:37.473 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":59,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:31:35.352818+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc6b806cc50 msgr2=0x7fc6b806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc6b806cc50 0x7fc6b806f110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc196b30 tx=0x7fc6bc03d040 comp rx=0 tx=0).stop 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 msgr2=0x7fc6cc19b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc19b6d0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400d6f0 tx=0x7fc6c400dbb0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 shutdown_connections 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc6b806cc50 0x7fc6b806f110 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc6cc104a30 0x7fc6cc19bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 --2- 172.21.15.90:0/923751637 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc6cc0fea30 0x7fc6cc19b6d0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:37.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 >> 172.21.15.90:0/923751637 conn(0x7fc6cc0fa730 msgr2=0x7fc6cc068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:37.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 shutdown_connections 2024-09-13T15:31:37.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:37.474+0000 7fc6d3113640 1 -- 172.21.15.90:0/923751637 wait complete. 2024-09-13T15:31:37.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:37 smithi090 ceph-mon[27811]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-09-13T15:31:37.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:37 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/923751637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:31:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:37 smithi175 ceph-mon[36128]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-09-13T15:31:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:37 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/923751637' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:31:38.136 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:31:38.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-13T15:31:38.335 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:39.746 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:39 smithi090 ceph-mon[27811]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T15:31:39.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:39 smithi175 ceph-mon[36128]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-13T15:31:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.943+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/95775812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 msgr2=0x7ffb28109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.943+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/95775812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb28109ac0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ffb10009880 tx=0x7ffb1002a530 comp rx=0 tx=0).stop 2024-09-13T15:31:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/95775812 shutdown_connections 2024-09-13T15:31:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/95775812 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb28109ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/95775812 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb28100cb0 0x7ffb28101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:39.947 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/95775812 >> 172.21.15.90:0/95775812 conn(0x7ffb280758a0 msgr2=0x7ffb280fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/95775812 shutdown_connections 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/95775812 wait complete. 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 Processor -- start 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.944+0000 7ffb2d2d4640 1 -- start start 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb2d2d4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb28100cb0 0x7ffb2819b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb2d2d4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb2819bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb2d2d4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2819c220 con 0x7ffb281015d0 2024-09-13T15:31:39.948 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb2d2d4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb28195780 con 0x7ffb28100cb0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb2819bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb2819bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47874/0 (socket says 172.21.15.90:47874) 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 -- 172.21.15.90:0/1516717065 learned_addr learned my addr 172.21.15.90:0/1516717065 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb26ffd640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb28100cb0 0x7ffb2819b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 -- 172.21.15.90:0/1516717065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb28100cb0 msgr2=0x7ffb2819b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb28100cb0 0x7ffb2819b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 -- 172.21.15.90:0/1516717065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb10009560 con 0x7ffb281015d0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb267fc640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb2819bbd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffb10006010 tx=0x7ffb10004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb100053e0 con 0x7ffb281015d0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb28195a00 con 0x7ffb281015d0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffb10005540 con 0x7ffb281015d0 2024-09-13T15:31:39.949 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.945+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffb1003c7c0 con 0x7ffb281015d0 2024-09-13T15:31:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.946+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb28195ef0 con 0x7ffb281015d0 2024-09-13T15:31:39.950 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.947+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb28068510 con 0x7ffb281015d0 2024-09-13T15:31:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.947+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ffb1002ad50 con 0x7ffb281015d0 2024-09-13T15:31:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.947+0000 7ffb1ffff640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ffb1406cc50 0x7ffb1406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.948+0000 7ffb26ffd640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ffb1406cc50 0x7ffb1406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.948+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7ffb100ad0c0 con 0x7ffb281015d0 2024-09-13T15:31:39.951 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.948+0000 7ffb26ffd640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ffb1406cc50 0x7ffb1406f110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb180058c0 tx=0x7ffb1800a2d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:39.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:39.952+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7ffb10077510 con 0x7ffb281015d0 2024-09-13T15:31:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.114+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffb28196c90 con 0x7ffb281015d0 2024-09-13T15:31:40.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.114+0000 7ffb1ffff640 1 -- 172.21.15.90:0/1516717065 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb10003b80 con 0x7ffb281015d0 2024-09-13T15:31:40.116 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ffb1406cc50 msgr2=0x7ffb1406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ffb1406cc50 0x7ffb1406f110 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffb180058c0 tx=0x7ffb1800a2d0 comp rx=0 tx=0).stop 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 msgr2=0x7ffb2819bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb2819bbd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffb10006010 tx=0x7ffb10004da0 comp rx=0 tx=0).stop 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 shutdown_connections 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ffb1406cc50 0x7ffb1406f110 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ffb281015d0 0x7ffb2819bbd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 --2- 172.21.15.90:0/1516717065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffb28100cb0 0x7ffb2819b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.116+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 >> 172.21.15.90:0/1516717065 conn(0x7ffb280758a0 msgr2=0x7ffb28101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.117+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 shutdown_connections 2024-09-13T15:31:40.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:40.117+0000 7ffb2d2d4640 1 -- 172.21.15.90:0/1516717065 wait complete. 2024-09-13T15:31:40.721 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:40 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1516717065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:31:40.722 INFO:tasks.cephadm:Setting up client nodes... 2024-09-13T15:31:40.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T15:31:40.921 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:41.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:40 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1516717065' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:31:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:41 smithi090 ceph-mon[27811]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-13T15:31:42.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:41 smithi175 ceph-mon[36128]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 -- 172.21.15.90:0/3694255627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240ffc70 msgr2=0x7f8c241000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 --2- 172.21.15.90:0/3694255627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240ffc70 0x7f8c241000f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8c18009880 tx=0x7f8c1802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 -- 172.21.15.90:0/3694255627 shutdown_connections 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 --2- 172.21.15.90:0/3694255627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240ffc70 0x7f8c241000f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 --2- 172.21.15.90:0/3694255627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c240fea70 0x7f8c240fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 -- 172.21.15.90:0/3694255627 >> 172.21.15.90:0/3694255627 conn(0x7f8c240fa200 msgr2=0x7f8c240fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 -- 172.21.15.90:0/3694255627 shutdown_connections 2024-09-13T15:31:42.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.559+0000 7f8c29f24640 1 -- 172.21.15.90:0/3694255627 wait complete. 2024-09-13T15:31:42.562 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c29f24640 1 Processor -- start 2024-09-13T15:31:42.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c29f24640 1 -- start start 2024-09-13T15:31:42.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c29f24640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 0x7f8c24195550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:42.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c29f24640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c240ffc70 0x7f8c24195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:42.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c29f24640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24195fd0 con 0x7f8c240fea70 2024-09-13T15:31:42.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c29f24640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24196140 con 0x7f8c240ffc70 2024-09-13T15:31:42.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.560+0000 7f8c237fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 0x7f8c24195550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c237fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 0x7f8c24195550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47892/0 (socket says 172.21.15.90:47892) 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c237fe640 1 -- 172.21.15.90:0/2765375271 learned_addr learned my addr 172.21.15.90:0/2765375271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c22ffd640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c240ffc70 0x7f8c24195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c237fe640 1 -- 172.21.15.90:0/2765375271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c240ffc70 msgr2=0x7f8c24195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c237fe640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c240ffc70 0x7f8c24195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c237fe640 1 -- 172.21.15.90:0/2765375271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18009560 con 0x7f8c240fea70 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c237fe640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 0x7f8c24195550 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400b3d0 tx=0x7f8c1400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c14011800 con 0x7f8c240fea70 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8c14011df0 con 0x7f8c240fea70 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8c1400f2e0 con 0x7f8c240fea70 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c2419ac20 con 0x7f8c240fea70 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.561+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c2419b1a0 con 0x7f8c240fea70 2024-09-13T15:31:42.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.562+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c24107530 con 0x7f8c240fea70 2024-09-13T15:31:42.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.563+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8c14011960 con 0x7f8c240fea70 2024-09-13T15:31:42.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.563+0000 7f8c20ff9640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8c0c06cc50 0x7f8c0c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:42.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.563+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f8c1408bf10 con 0x7f8c240fea70 2024-09-13T15:31:42.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.563+0000 7f8c22ffd640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8c0c06cc50 0x7f8c0c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:42.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.564+0000 7f8c22ffd640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8c0c06cc50 0x7f8c0c06f110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c18004f60 tx=0x7f8c18000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:42.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.567+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f8c14056360 con 0x7f8c240fea70 2024-09-13T15:31:42.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.796+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f8c2419b420 con 0x7f8c240fea70 2024-09-13T15:31:42.801 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.800+0000 7f8c20ff9640 1 -- 172.21.15.90:0/2765375271 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (secure 0 0 0) 0x7f8c14015050 con 0x7f8c240fea70 2024-09-13T15:31:42.801 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2024-09-13T15:31:42.801 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQDeWuRmOIeJLxAA4+0NoAuoqY73UCKzYuOO6Q== 2024-09-13T15:31:42.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8c0c06cc50 msgr2=0x7f8c0c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:42.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8c0c06cc50 0x7f8c0c06f110 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8c18004f60 tx=0x7f8c18000bc0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 msgr2=0x7f8c24195550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:42.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 0x7f8c24195550 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8c1400b3d0 tx=0x7f8c1400b890 comp rx=0 tx=0).stop 2024-09-13T15:31:42.803 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 shutdown_connections 2024-09-13T15:31:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8c0c06cc50 0x7f8c0c06f110 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c240ffc70 0x7f8c24195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 --2- 172.21.15.90:0/2765375271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8c240fea70 0x7f8c24195550 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 >> 172.21.15.90:0/2765375271 conn(0x7f8c240fa200 msgr2=0x7f8c24067c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 shutdown_connections 2024-09-13T15:31:42.804 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:42.802+0000 7f8c29f24640 1 -- 172.21.15.90:0/2765375271 wait complete. 2024-09-13T15:31:43.498 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:31:43.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-13T15:31:43.499 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T15:31:43.535 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-13T15:31:43.747 INFO:teuthology.orchestra.run.smithi175.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi175/config 2024-09-13T15:31:43.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:43 smithi090 ceph-mon[27811]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T15:31:43.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:43 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2765375271' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T15:31:43.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:43 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2765375271' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T15:31:43.867 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:43 smithi175 ceph-mon[36128]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-13T15:31:43.867 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:43 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2765375271' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T15:31:43.867 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:43 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2765375271' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T15:31:45.565 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 -- 172.21.15.175:0/3276188335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00fea80 msgr2=0x7f19f00fee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:45.565 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 --2- 172.21.15.175:0/3276188335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00fea80 0x7f19f00fee80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009880 tx=0x7f19e002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 -- 172.21.15.175:0/3276188335 shutdown_connections 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 --2- 172.21.15.175:0/3276188335 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19f00ffc80 0x7f19f0100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 --2- 172.21.15.175:0/3276188335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00fea80 0x7f19f00fee80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 -- 172.21.15.175:0/3276188335 >> 172.21.15.175:0/3276188335 conn(0x7f19f00fa230 msgr2=0x7f19f00fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 -- 172.21.15.175:0/3276188335 shutdown_connections 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.561+0000 7f19f61ae640 1 -- 172.21.15.175:0/3276188335 wait complete. 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.562+0000 7f19f61ae640 1 Processor -- start 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.562+0000 7f19f61ae640 1 -- start start 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.562+0000 7f19f61ae640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19f00fea80 0x7f19f0068fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.562+0000 7f19f61ae640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 0x7f19f019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.562+0000 7f19f61ae640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f019a740 con 0x7f19f00fea80 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.562+0000 7f19f61ae640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f019a880 con 0x7f19f00ffc80 2024-09-13T15:31:45.566 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 0x7f19f019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 0x7f19f019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:47880/0 (socket says 172.21.15.175:47880) 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 -- 172.21.15.175:0/847057493 learned_addr learned my addr 172.21.15.175:0/847057493 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19ef7fe640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19f00fea80 0x7f19f0068fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 -- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19f00fea80 msgr2=0x7f19f0068fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19f00fea80 0x7f19f0068fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 -- 172.21.15.175:0/847057493 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e0009560 con 0x7f19f00ffc80 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19eeffd640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 0x7f19f019a200 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f19e400d6f0 tx=0x7f19e400dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19e4009a30 con 0x7f19f00ffc80 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f19e40054b0 con 0x7f19f00ffc80 2024-09-13T15:31:45.567 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f019ab30 con 0x7f19f00ffc80 2024-09-13T15:31:45.568 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19e4004d00 con 0x7f19f00ffc80 2024-09-13T15:31:45.568 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.563+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f019b100 con 0x7f19f00ffc80 2024-09-13T15:31:45.568 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.564+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f01074c0 con 0x7f19f00ffc80 2024-09-13T15:31:45.568 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.565+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f19e4004f70 con 0x7f19f00ffc80 2024-09-13T15:31:45.569 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.565+0000 7f19ecff9640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f19d806cc50 0x7f19d806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:45.569 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.565+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f19e408bce0 con 0x7f19f00ffc80 2024-09-13T15:31:45.569 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.566+0000 7f19ef7fe640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f19d806cc50 0x7f19d806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:45.569 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.566+0000 7f19ef7fe640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f19d806cc50 0x7f19d806f110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009230 tx=0x7f19e0004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:45.572 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.570+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f19e4056130 con 0x7f19f00ffc80 2024-09-13T15:31:45.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:45 smithi175 ceph-mon[36128]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:31:45.794 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.792+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f19f01076d0 con 0x7f19f00ffc80 2024-09-13T15:31:45.799 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.797+0000 7f19ecff9640 1 -- 172.21.15.175:0/847057493 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v19) v1 ==== 170+0+59 (secure 0 0 0) 0x7f19e40170f0 con 0x7f19f00ffc80 2024-09-13T15:31:45.800 INFO:teuthology.orchestra.run.smithi175.stdout:[client.1] 2024-09-13T15:31:45.800 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQDhWuRmBOJmLxAAjD6kja7b4cTsRCdxHsv9PQ== 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f19d806cc50 msgr2=0x7f19d806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f19d806cc50 0x7f19d806f110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f19e0009230 tx=0x7f19e0004f00 comp rx=0 tx=0).stop 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 msgr2=0x7f19f019a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 0x7f19f019a200 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f19e400d6f0 tx=0x7f19e400dbb0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 shutdown_connections 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f19d806cc50 0x7f19d806f110 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.802 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f00ffc80 0x7f19f019a200 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 --2- 172.21.15.175:0/847057493 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f19f00fea80 0x7f19f0068fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:45.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 >> 172.21.15.175:0/847057493 conn(0x7f19f00fa230 msgr2=0x7f19f0102eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:45.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 shutdown_connections 2024-09-13T15:31:45.803 INFO:teuthology.orchestra.run.smithi175.stderr:2024-09-13T15:31:45.800+0000 7f19f61ae640 1 -- 172.21.15.175:0/847057493 wait complete. 2024-09-13T15:31:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:45 smithi090 ceph-mon[27811]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-13T15:31:46.477 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:31:46.477 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-13T15:31:46.477 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T15:31:46.515 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T15:31:46.516 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-13T15:31:46.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-13T15:31:46.726 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:46.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.175:0/847057493' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T15:31:46.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:46 smithi175 ceph-mon[36128]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T15:31:46.766 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:46 smithi175 ceph-mon[36128]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T15:31:46.847 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.175:0/847057493' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T15:31:46.847 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:46 smithi090 ceph-mon[27811]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-13T15:31:46.847 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:46 smithi090 ceph-mon[27811]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-13T15:31:48.087 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:47 smithi090 ceph-mon[27811]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-13T15:31:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:47 smithi175 ceph-mon[36128]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-13T15:31:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/1243256175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 msgr2=0x7fc4a40fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/1243256175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a40fee50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc498009880 tx=0x7fc49802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/1243256175 shutdown_connections 2024-09-13T15:31:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/1243256175 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a40fee50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.328 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/1243256175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 0x7fc4a4104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/1243256175 >> 172.21.15.90:0/1243256175 conn(0x7fc4a40758a0 msgr2=0x7fc4a40fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/1243256175 shutdown_connections 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/1243256175 wait complete. 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 Processor -- start 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.314+0000 7fc4ab7ae640 1 -- start start 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4ab7ae640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a419b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4ab7ae640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 0x7fc4a419bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4ab7ae640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a419c1f0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4ab7ae640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a419c360 con 0x7fc4a41049f0 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a419b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a419b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35498/0 (socket says 172.21.15.90:35498) 2024-09-13T15:31:48.329 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 -- 172.21.15.90:0/2513906584 learned_addr learned my addr 172.21.15.90:0/2513906584 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a8d49640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 0x7fc4a419bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 -- 172.21.15.90:0/2513906584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 msgr2=0x7fc4a419bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 0x7fc4a419bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 -- 172.21.15.90:0/2513906584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc498009560 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a8d49640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 0x7fc4a419bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a954a640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a419b6b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e970 tx=0x7fc49400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc49400cc90 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc49400cdf0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a41959a0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc494004e50 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.315+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a4195ec0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.330 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.316+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a4100130 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.317+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc494005020 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.317+0000 7fc4a27fc640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc48c06cc50 0x7fc48c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.317+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fc494014070 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.317+0000 7fc4a8d49640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc48c06cc50 0x7fc48c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.318+0000 7fc4a8d49640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc48c06cc50 0x7fc48c06f110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4197060 tx=0x7fc498005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:48.331 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.322+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fc494056ba0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.528+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc4a410bab0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.531+0000 7fc4a27fc640 1 -- 172.21.15.90:0/2513906584 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+174009 (secure 0 0 0) 0x7fc4940052d0 con 0x7fc4a40fe9f0 2024-09-13T15:31:48.535 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:31:48.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.533+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc48c06cc50 msgr2=0x7fc48c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.533+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc48c06cc50 0x7fc48c06f110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4197060 tx=0x7fc498005a20 comp rx=0 tx=0).stop 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.533+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 msgr2=0x7fc4a419b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.533+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a419b6b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e970 tx=0x7fc49400ed70 comp rx=0 tx=0).stop 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 shutdown_connections 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc48c06cc50 0x7fc48c06f110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc4a41049f0 0x7fc4a419bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 --2- 172.21.15.90:0/2513906584 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc4a40fe9f0 0x7fc4a419b6b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 >> 172.21.15.90:0/2513906584 conn(0x7fc4a40758a0 msgr2=0x7fc4a40fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 shutdown_connections 2024-09-13T15:31:48.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:48.534+0000 7fc4ab7ae640 1 -- 172.21.15.90:0/2513906584 wait complete. 2024-09-13T15:31:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:48 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2513906584' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T15:31:49.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:49.118 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:48 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2513906584' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-13T15:31:49.118 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":19,"flags":0,"active_gid":14217,"active_name":"smithi090.fyvvpd","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":885641711},{"type":"v1","addr":"172.21.15.90:6801","nonce":885641711}]},"active_addr":"172.21.15.90:6801/885641711","active_change":"2024-09-13T15:27:18.494518+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":14240,"name":"smithi175.otzykk","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.90:8443/","prometheus":"http://172.21.15.90:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2692969918}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":225534373}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":2929560719}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1980891070}]}]} 2024-09-13T15:31:49.144 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-13T15:31:49.144 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-13T15:31:49.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:31:49.343 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:49 smithi090 ceph-mon[27811]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:49 smithi175 ceph-mon[36128]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:50.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.810+0000 7f38ff420640 1 -- 172.21.15.90:0/3031261238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f8105c00 msgr2=0x7f38f8105fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.810+0000 7f38ff420640 1 --2- 172.21.15.90:0/3031261238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f8105c00 0x7f38f8105fe0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009880 tx=0x7f38f002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.810+0000 7f38ff420640 1 -- 172.21.15.90:0/3031261238 shutdown_connections 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.810+0000 7f38ff420640 1 --2- 172.21.15.90:0/3031261238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f80ffb50 0x7f38f80fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.810+0000 7f38ff420640 1 --2- 172.21.15.90:0/3031261238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f8105c00 0x7f38f8105fe0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.810+0000 7f38ff420640 1 -- 172.21.15.90:0/3031261238 >> 172.21.15.90:0/3031261238 conn(0x7f38f80fb830 msgr2=0x7f38f80fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 -- 172.21.15.90:0/3031261238 shutdown_connections 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 -- 172.21.15.90:0/3031261238 wait complete. 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 Processor -- start 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 -- start start 2024-09-13T15:31:50.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 0x7f38f80727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f8105c00 0x7f38f8072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8074780 con 0x7f38f80ffb50 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.811+0000 7f38ff420640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f8073230 con 0x7f38f8105c00 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 0x7f38f80727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 0x7f38f80727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35512/0 (socket says 172.21.15.90:35512) 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 -- 172.21.15.90:0/2702062238 learned_addr learned my addr 172.21.15.90:0/2702062238 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fc9bb640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f8105c00 0x7f38f8072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 -- 172.21.15.90:0/2702062238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f8105c00 msgr2=0x7f38f8072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f8105c00 0x7f38f8072cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 -- 172.21.15.90:0/2702062238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f0009560 con 0x7f38f80ffb50 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fc9bb640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f8105c00 0x7f38f8072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38fd1bc640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 0x7f38f80727b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006010 tx=0x7f38f0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:50.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f0005460 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f80734b0 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f38f00055c0 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f38f003c900 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.812+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f81a1d80 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.813+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38f8101200 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.814+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f38f0003c10 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.814+0000 7f38f67fc640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f38e006cc50 0x7f38e006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.814+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f38f00ad520 con 0x7f38f80ffb50 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.814+0000 7f38fc9bb640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f38e006cc50 0x7f38e006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.814+0000 7f38fc9bb640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f38e006cc50 0x7f38e006f110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f38ec005fd0 tx=0x7f38ec005ea0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:50.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.818+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f38f0077840 con 0x7f38f80ffb50 2024-09-13T15:31:50.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.985+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38f8073d80 con 0x7f38f80ffb50 2024-09-13T15:31:50.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.986+0000 7f38f67fc640 1 -- 172.21.15.90:0/2702062238 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13848 (secure 0 0 0) 0x7f38f007ae60 con 0x7f38f80ffb50 2024-09-13T15:31:50.988 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:31:50.988 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":59,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:31:35.352818+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:31:50.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f38e006cc50 msgr2=0x7f38e006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:50.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f38e006cc50 0x7f38e006f110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f38ec005fd0 tx=0x7f38ec005ea0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 msgr2=0x7f38f80727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:50.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 0x7f38f80727b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006010 tx=0x7f38f0004610 comp rx=0 tx=0).stop 2024-09-13T15:31:50.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 shutdown_connections 2024-09-13T15:31:50.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f38e006cc50 0x7f38e006f110 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f38f8105c00 0x7f38f8072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 --2- 172.21.15.90:0/2702062238 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f38f80ffb50 0x7f38f80727b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:50.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 >> 172.21.15.90:0/2702062238 conn(0x7f38f80fb830 msgr2=0x7f38f80fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:50.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 shutdown_connections 2024-09-13T15:31:50.992 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:50.990+0000 7f38ff420640 1 -- 172.21.15.90:0/2702062238 wait complete. 2024-09-13T15:31:51.562 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-13T15:31:51.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-13T15:31:51.761 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:52.109 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:51 smithi090 ceph-mon[27811]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:52.110 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:51 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2702062238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:31:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:51 smithi175 ceph-mon[36128]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:51 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2702062238' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3011302192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 msgr2=0x7fb0e8109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3011302192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e8109b00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0098b0 tx=0x7fb0dc02a4d0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3011302192 shutdown_connections 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3011302192 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e8109b00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3011302192 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0e8100c60 0x7fb0e8101040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3011302192 >> 172.21.15.90:0/3011302192 conn(0x7fb0e80fa730 msgr2=0x7fb0e80fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:53.335 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3011302192 shutdown_connections 2024-09-13T15:31:53.336 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.333+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3011302192 wait complete. 2024-09-13T15:31:53.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.334+0000 7fb0ee3ab640 1 Processor -- start 2024-09-13T15:31:53.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.334+0000 7fb0ee3ab640 1 -- start start 2024-09-13T15:31:53.337 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0ee3ab640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0e8100c60 0x7fb0e819b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0ee3ab640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0ee3ab640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e819c2d0 con 0x7fb0e8101610 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0ee3ab640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81958f0 con 0x7fb0e8100c60 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e819bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:35536/0 (socket says 172.21.15.90:35536) 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 -- 172.21.15.90:0/3821479595 learned_addr learned my addr 172.21.15.90:0/3821479595 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e7fff640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0e8100c60 0x7fb0e819b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 -- 172.21.15.90:0/3821479595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0e8100c60 msgr2=0x7fb0e819b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0e8100c60 0x7fb0e819b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.338 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 -- 172.21.15.90:0/3821479595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc009560 con 0x7fb0e8101610 2024-09-13T15:31:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e77fe640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e819bd40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0099d0 tx=0x7fb0dc005130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0dc005680 con 0x7fb0e8101610 2024-09-13T15:31:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8195b70 con 0x7fb0e8101610 2024-09-13T15:31:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb0dc004d40 con 0x7fb0e8101610 2024-09-13T15:31:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.335+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0dc03c860 con 0x7fb0e8101610 2024-09-13T15:31:53.339 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.336+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8196060 con 0x7fb0e8101610 2024-09-13T15:31:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.337+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8068d30 con 0x7fb0e8101610 2024-09-13T15:31:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.337+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb0dc0035f0 con 0x7fb0e8101610 2024-09-13T15:31:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.338+0000 7fb0e57fa640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb0d006cc50 0x7fb0d006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:31:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.338+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fb0dc0ad540 con 0x7fb0e8101610 2024-09-13T15:31:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.338+0000 7fb0e7fff640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb0d006cc50 0x7fb0d006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:31:53.340 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.338+0000 7fb0e7fff640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb0d006cc50 0x7fb0d006f110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009990 tx=0x7fb0d8009110 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:31:53.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.342+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb0dc077860 con 0x7fb0e8101610 2024-09-13T15:31:53.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.535+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0e81010c0 con 0x7fb0e8101610 2024-09-13T15:31:53.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.536+0000 7fb0e57fa640 1 -- 172.21.15.90:0/3821479595 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13848 (secure 0 0 0) 0x7fb0dc07ae80 con 0x7fb0e8101610 2024-09-13T15:31:53.538 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:31:53.538 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":59,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:31:35.352818+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb0d006cc50 msgr2=0x7fb0d006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb0d006cc50 0x7fb0d006f110 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009990 tx=0x7fb0d8009110 comp rx=0 tx=0).stop 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 msgr2=0x7fb0e819bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e819bd40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc0099d0 tx=0x7fb0dc005130 comp rx=0 tx=0).stop 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 shutdown_connections 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb0d006cc50 0x7fb0d006f110 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb0e8101610 0x7fb0e819bd40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.538+0000 7fb0ee3ab640 1 --2- 172.21.15.90:0/3821479595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0e8100c60 0x7fb0e819b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:31:53.540 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.539+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 >> 172.21.15.90:0/3821479595 conn(0x7fb0e80fa730 msgr2=0x7fb0e8101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:31:53.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.539+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 shutdown_connections 2024-09-13T15:31:53.541 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:31:53.539+0000 7fb0ee3ab640 1 -- 172.21.15.90:0/3821479595 wait complete. 2024-09-13T15:31:54.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:53 smithi090 ceph-mon[27811]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:54.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:53 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3821479595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:31:54.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-13T15:31:54.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-13T15:31:54.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-13T15:31:54.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-13T15:31:54.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-13T15:31:54.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-13T15:31:54.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-13T15:31:54.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-13T15:31:54.269 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:53 smithi175 ceph-mon[36128]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:54.269 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:53 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3821479595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:31:54.731 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.732 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.736 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.736 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.737 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.738 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.742 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:54.743 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:31:56.077 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:55 smithi090 ceph-mon[27811]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:55 smithi175 ceph-mon[36128]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:57.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:57 smithi090 ceph-mon[27811]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:58.244 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:57 smithi175 ceph-mon[36128]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:31:59.990 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:31:59 smithi090 ceph-mon[27811]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:31:59 smithi175 ceph-mon[36128]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:02.234 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:01 smithi090 ceph-mon[27811]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:01 smithi175 ceph-mon[36128]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:03.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 -- 172.21.15.90:0/4157897627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 msgr2=0x7f696c0fee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:03.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 --2- 172.21.15.90:0/4157897627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c0fee70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f695c009880 tx=0x7f695c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 -- 172.21.15.90:0/4157897627 shutdown_connections 2024-09-13T15:32:03.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 --2- 172.21.15.90:0/4157897627 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c0fee70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 --2- 172.21.15.90:0/4157897627 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f696c104a10 0x7f696c104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 -- 172.21.15.90:0/4157897627 >> 172.21.15.90:0/4157897627 conn(0x7f696c0fa730 msgr2=0x7f696c0fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 -- 172.21.15.90:0/4157897627 shutdown_connections 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.963+0000 7f6973e99640 1 -- 172.21.15.90:0/4157897627 wait complete. 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6973e99640 1 Processor -- start 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6973e99640 1 -- start start 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6973e99640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c19b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6973e99640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f696c104a10 0x7f696c19bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6973e99640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c19c260 con 0x7f696c0fea10 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6973e99640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c195880 con 0x7f696c104a10 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6971c35640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c19b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6971c35640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c19b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41234/0 (socket says 172.21.15.90:41234) 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.964+0000 7f6971c35640 1 -- 172.21.15.90:0/77385048 learned_addr learned my addr 172.21.15.90:0/77385048 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:03.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6971434640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f696c104a10 0x7f696c19bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6971c35640 1 -- 172.21.15.90:0/77385048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f696c104a10 msgr2=0x7f696c19bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6971c35640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f696c104a10 0x7f696c19bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6971c35640 1 -- 172.21.15.90:0/77385048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c009560 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6971c35640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c19b790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f696800b420 tx=0x7f696800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6968011800 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c195b60 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6968011df0 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6968010e80 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.965+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c1960b0 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.966+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f696c10be30 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.966+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6968011960 con 0x7f696c0fea10 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6962ffd640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f695806cc50 0x7f695806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:03.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f696808bc80 con 0x7f696c0fea10 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6971434640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f695806cc50 0x7f695806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6962ffd640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] conn(0x7f6958072fb0 0x7f69580753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 --> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6958075a80 con 0x7f6958072fb0 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6968091360 con 0x7f696c0fea10 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6972436640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] conn(0x7f6958072fb0 0x7f69580753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6971434640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f695806cc50 0x7f695806f110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f695c006010 tx=0x7f695c000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.967+0000 7f6972436640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] conn(0x7f6958072fb0 0x7f69580753d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.968+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== osd.1 v2:172.21.15.90:6810/3388650116 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f6958075a80 con 0x7f6958072fb0 2024-09-13T15:32:03.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 --> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f696c100150 con 0x7f6958072fb0 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6962ffd640 1 -- 172.21.15.90:0/77385048 <== osd.1 v2:172.21.15.90:6810/3388650116 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f696c100150 con 0x7f6958072fb0 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] conn(0x7f6958072fb0 msgr2=0x7f69580753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] conn(0x7f6958072fb0 0x7f69580753d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f695806cc50 msgr2=0x7f695806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f695806cc50 0x7f695806f110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f695c006010 tx=0x7f695c000bc0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 msgr2=0x7f696c19b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c19b790 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f696800b420 tx=0x7f696800b8e0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 shutdown_connections 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6810/3388650116,v1:172.21.15.90:6811/3388650116] conn(0x7f6958072fb0 0x7f69580753d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f695806cc50 0x7f695806f110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f696c104a10 0x7f696c19bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 --2- 172.21.15.90:0/77385048 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f696c0fea10 0x7f696c19b790 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:03.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 >> 172.21.15.90:0/77385048 conn(0x7f696c0fa730 msgr2=0x7f696c068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:03.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 shutdown_connections 2024-09-13T15:32:03.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:01.981+0000 7f6973e99640 1 -- 172.21.15.90:0/77385048 wait complete. 2024-09-13T15:32:04.133 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:03 smithi090 ceph-mon[27811]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:04.133 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 -- 172.21.15.90:0/2114090431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9604104a10 msgr2=0x7f9604104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 --2- 172.21.15.90:0/2114090431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9604104a10 0x7f9604104df0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f95f8009880 tx=0x7f95f802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 -- 172.21.15.90:0/2114090431 shutdown_connections 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 --2- 172.21.15.90:0/2114090431 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96040fea10 0x7f96040fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 --2- 172.21.15.90:0/2114090431 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9604104a10 0x7f9604104df0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 -- 172.21.15.90:0/2114090431 >> 172.21.15.90:0/2114090431 conn(0x7f96040fa730 msgr2=0x7f96040fcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 -- 172.21.15.90:0/2114090431 shutdown_connections 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.212+0000 7f960b154640 1 -- 172.21.15.90:0/2114090431 wait complete. 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f960b154640 1 Processor -- start 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f960b154640 1 -- start start 2024-09-13T15:32:04.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f960b154640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 0x7f960419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f960b154640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9604104a10 0x7f960419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f960b154640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960419b0a0 con 0x7f96040fea10 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f960b154640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96041946c0 con 0x7f9604104a10 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f9608ef0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 0x7f960419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f9608ef0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 0x7f960419a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:41248/0 (socket says 172.21.15.90:41248) 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.213+0000 7f9608ef0640 1 -- 172.21.15.90:0/4147031075 learned_addr learned my addr 172.21.15.90:0/4147031075 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9603fff640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9604104a10 0x7f960419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9608ef0640 1 -- 172.21.15.90:0/4147031075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9604104a10 msgr2=0x7f960419ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9608ef0640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9604104a10 0x7f960419ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9608ef0640 1 -- 172.21.15.90:0/4147031075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f8009560 con 0x7f96040fea10 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9608ef0640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 0x7f960419a5d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f8004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95f8005460 con 0x7f96040fea10 2024-09-13T15:32:04.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f95f80055c0 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9604194940 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95f803c900 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.214+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9604194e30 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.215+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f960410bdd0 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f95f8003c10 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9601ffb640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f95f406cc50 0x7f95f406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f95f80ad550 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9603fff640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f95f406cc50 0x7f95f406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9601ffb640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] conn(0x7f95f4072fb0 0x7f95f40753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 --> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95f4075a80 con 0x7f95f4072fb0 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95f80afc20 con 0x7f96040fea10 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.216+0000 7f96096f1640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] conn(0x7f95f4072fb0 0x7f95f40753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.217+0000 7f9603fff640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f95f406cc50 0x7f95f406f110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95f0005e10 tx=0x7f95f0005da0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.217+0000 7f96096f1640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] conn(0x7f95f4072fb0 0x7f95f40753d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.217+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== osd.3 v2:172.21.15.90:6826/1255290974 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f95f4075a80 con 0x7f95f4072fb0 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.230+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 --> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9604100150 con 0x7f95f4072fb0 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.230+0000 7f9601ffb640 1 -- 172.21.15.90:0/4147031075 <== osd.3 v2:172.21.15.90:6826/1255290974 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9604100150 con 0x7f95f4072fb0 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] conn(0x7f95f4072fb0 msgr2=0x7f95f40753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] conn(0x7f95f4072fb0 0x7f95f40753d0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f95f406cc50 msgr2=0x7f95f406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f95f406cc50 0x7f95f406f110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f95f0005e10 tx=0x7f95f0005da0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 msgr2=0x7f960419a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 0x7f960419a5d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f95f8006010 tx=0x7f95f8004610 comp rx=0 tx=0).stop 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 shutdown_connections 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6826/1255290974,v1:172.21.15.90:6827/1255290974] conn(0x7f95f4072fb0 0x7f95f40753d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f95f406cc50 0x7f95f406f110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9604104a10 0x7f960419ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 --2- 172.21.15.90:0/4147031075 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f96040fea10 0x7f960419a5d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 >> 172.21.15.90:0/4147031075 conn(0x7f96040fa730 msgr2=0x7f96040fb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 shutdown_connections 2024-09-13T15:32:04.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.231+0000 7f960b154640 1 -- 172.21.15.90:0/4147031075 wait complete. 2024-09-13T15:32:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:03 smithi175 ceph-mon[36128]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:04.389 INFO:teuthology.orchestra.run.smithi090.stdout:60129542183 2024-09-13T15:32:04.390 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 -- 172.21.15.90:0/4259312227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 msgr2=0x7fb8e81047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 --2- 172.21.15.90:0/4259312227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e81047a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4009880 tx=0x7fb8d402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 -- 172.21.15.90:0/4259312227 shutdown_connections 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 --2- 172.21.15.90:0/4259312227 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e81047a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 --2- 172.21.15.90:0/4259312227 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 0x7fb8e8100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 -- 172.21.15.90:0/4259312227 >> 172.21.15.90:0/4259312227 conn(0x7fb8e80fb840 msgr2=0x7fb8e80fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:04.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 -- 172.21.15.90:0/4259312227 shutdown_connections 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.749+0000 7fb8eee64640 1 -- 172.21.15.90:0/4259312227 wait complete. 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8eee64640 1 Processor -- start 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8eee64640 1 -- start start 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8eee64640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 0x7fb8e819b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8eee64640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e819bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8eee64640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e819c270 con 0x7fb8e8100750 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8eee64640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81957d0 con 0x7fb8e80ffda0 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8e7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e819bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8e7fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e819bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51416/0 (socket says 172.21.15.90:51416) 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8e7fff640 1 -- 172.21.15.90:0/3412691280 learned_addr learned my addr 172.21.15.90:0/3412691280 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.750+0000 7fb8ecc00640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 0x7fb8e819b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e7fff640 1 -- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 msgr2=0x7fb8e819b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e7fff640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 0x7fb8e819b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e7fff640 1 -- 172.21.15.90:0/3412691280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d4009560 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8ecc00640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 0x7fb8e819b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e7fff640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e819bc20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4006010 tx=0x7fb8d4004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d4005460 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb8d40055c0 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e8195a50 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d403c900 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.751+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e8195f40 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.752+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8e8101280 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.752+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb8d4003c10 con 0x7fb8e8100750 2024-09-13T15:32:04.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.753+0000 7fb8e5ffb640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb8d806cc50 0x7fb8d806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.753+0000 7fb8ecc00640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb8d806cc50 0x7fb8d806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.753+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fb8d40ad550 con 0x7fb8e8100750 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.753+0000 7fb8e5ffb640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] conn(0x7fb8d8072fb0 0x7fb8d80753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.753+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 --> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8d8075a80 con 0x7fb8d8072fb0 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.753+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8d40afc20 con 0x7fb8e8100750 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.754+0000 7fb8ecc00640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb8d806cc50 0x7fb8d806f110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00abe0 tx=0x7fb8dc00a5c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.754+0000 7fb8ed401640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] conn(0x7fb8d8072fb0 0x7fb8d80753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.754+0000 7fb8ed401640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] conn(0x7fb8d8072fb0 0x7fb8d80753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:04.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.755+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== osd.5 v2:172.21.15.175:6808/1093936087 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb8d8075a80 con 0x7fb8d8072fb0 2024-09-13T15:32:04.769 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.768+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 --> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb8e810ce20 con 0x7fb8d8072fb0 2024-09-13T15:32:04.770 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8e5ffb640 1 -- 172.21.15.90:0/3412691280 <== osd.5 v2:172.21.15.175:6808/1093936087 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb8e810ce20 con 0x7fb8d8072fb0 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] conn(0x7fb8d8072fb0 msgr2=0x7fb8d80753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] conn(0x7fb8d8072fb0 0x7fb8d80753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb8d806cc50 msgr2=0x7fb8d806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb8d806cc50 0x7fb8d806f110 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc00abe0 tx=0x7fb8dc00a5c0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 msgr2=0x7fb8e819bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e819bc20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4006010 tx=0x7fb8d4004610 comp rx=0 tx=0).stop 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 shutdown_connections 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb8d806cc50 0x7fb8d806f110 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb8e8100750 0x7fb8e819bc20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:6808/1093936087,v1:172.21.15.175:6809/1093936087] conn(0x7fb8d8072fb0 0x7fb8d80753d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.771 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 --2- 172.21.15.90:0/3412691280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb8e80ffda0 0x7fb8e819b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 >> 172.21.15.90:0/3412691280 conn(0x7fb8e80fb840 msgr2=0x7fb8e80fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 shutdown_connections 2024-09-13T15:32:04.772 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:04.769+0000 7fb8eee64640 1 -- 172.21.15.90:0/3412691280 wait complete. 2024-09-13T15:32:06.162 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:05 smithi090 ceph-mon[27811]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout:167503724563 2024-09-13T15:32:06.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-13T15:32:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:05 smithi175 ceph-mon[36128]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:06.363 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.340+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/566033319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b4100750 msgr2=0x7fd4b41047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.340+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/566033319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b4100750 0x7fd4b41047a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd4a8009880 tx=0x7fd4a802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/566033319 shutdown_connections 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/566033319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b4100750 0x7fd4b41047a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/566033319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4b40ffda0 0x7fd4b4100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/566033319 >> 172.21.15.90:0/566033319 conn(0x7fd4b40fb840 msgr2=0x7fd4b40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/566033319 shutdown_connections 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/566033319 wait complete. 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 Processor -- start 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.341+0000 7fd4b8e1f640 1 -- start start 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b8e1f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 0x7fd4b419b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:06.364 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b8e1f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4b4100750 0x7fd4b419bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b8e1f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b419c2d0 con 0x7fd4b40ffda0 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b8e1f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b41958f0 con 0x7fd4b4100750 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 0x7fd4b419b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 0x7fd4b419b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51434/0 (socket says 172.21.15.90:51434) 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 -- 172.21.15.90:0/3383727986 learned_addr learned my addr 172.21.15.90:0/3383727986 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b1d9b640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4b4100750 0x7fd4b419bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 -- 172.21.15.90:0/3383727986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4b4100750 msgr2=0x7fd4b419bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4b4100750 0x7fd4b419bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 -- 172.21.15.90:0/3383727986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a8009560 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b259c640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 0x7fd4b419b800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b420 tx=0x7fd4a400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a4011800 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b4195bd0 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4a4011df0 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4a4010e80 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.342+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b4196120 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.343+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4b410cf60 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.344+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd4a4011960 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.344+0000 7fd4a37fe640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd49c06cc50 0x7fd49c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fd4a408c6c0 con 0x7fd4b40ffda0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4b1d9b640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd49c06cc50 0x7fd49c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:06.366 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4a37fe640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] conn(0x7fd49c072fb0 0x7fd49c0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 --> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd49c075a80 con 0x7fd49c072fb0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4a408cb20 con 0x7fd4b40ffda0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4b2d9d640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] conn(0x7fd49c072fb0 0x7fd49c0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4b1d9b640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd49c06cc50 0x7fd49c06f110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4196d60 tx=0x7fd4a8005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.345+0000 7fd4b2d9d640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] conn(0x7fd49c072fb0 0x7fd49c0753d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.346+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== osd.2 v2:172.21.15.90:6818/3155836791 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fd49c075a80 con 0x7fd49c072fb0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 --> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4b41965e0 con 0x7fd49c072fb0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4a37fe640 1 -- 172.21.15.90:0/3383727986 <== osd.2 v2:172.21.15.90:6818/3155836791 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd4b41965e0 con 0x7fd49c072fb0 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] conn(0x7fd49c072fb0 msgr2=0x7fd49c0753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] conn(0x7fd49c072fb0 0x7fd49c0753d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd49c06cc50 msgr2=0x7fd49c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:06.367 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd49c06cc50 0x7fd49c06f110 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4196d60 tx=0x7fd4a8005a20 comp rx=0 tx=0).stop 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 msgr2=0x7fd4b419b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 0x7fd4b419b800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd4a400b420 tx=0x7fd4a400b8e0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 shutdown_connections 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6818/3155836791,v1:172.21.15.90:6819/3155836791] conn(0x7fd49c072fb0 0x7fd49c0753d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd49c06cc50 0x7fd49c06f110 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4b4100750 0x7fd4b419bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 --2- 172.21.15.90:0/3383727986 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4b40ffda0 0x7fd4b419b800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 >> 172.21.15.90:0/3383727986 conn(0x7fd4b40fb840 msgr2=0x7fd4b40fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 shutdown_connections 2024-09-13T15:32:06.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:06.359+0000 7fd4b8e1f640 1 -- 172.21.15.90:0/3383727986 wait complete. 2024-09-13T15:32:06.823 INFO:teuthology.orchestra.run.smithi090.stdout:111669149724 2024-09-13T15:32:06.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.334+0000 7f5982423640 1 -- 172.21.15.90:0/662977590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c100750 msgr2=0x7f597c1047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.334+0000 7f5982423640 1 --2- 172.21.15.90:0/662977590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c100750 0x7f597c1047a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5970009880 tx=0x7f597002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 -- 172.21.15.90:0/662977590 shutdown_connections 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 --2- 172.21.15.90:0/662977590 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c100750 0x7f597c1047a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 --2- 172.21.15.90:0/662977590 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f597c0ffda0 0x7f597c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 -- 172.21.15.90:0/662977590 >> 172.21.15.90:0/662977590 conn(0x7f597c0fb840 msgr2=0x7f597c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:07.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 -- 172.21.15.90:0/662977590 shutdown_connections 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 -- 172.21.15.90:0/662977590 wait complete. 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 Processor -- start 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.335+0000 7f5982423640 1 -- start start 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f5982423640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 0x7f597c19b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f5982423640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f597c100750 0x7f597c19bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f5982423640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c19c2d0 con 0x7f597c0ffda0 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f5982423640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597c1958f0 con 0x7f597c100750 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 0x7f597c19b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 0x7f597c19b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51458/0 (socket says 172.21.15.90:51458) 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 -- 172.21.15.90:0/4067064233 learned_addr learned my addr 172.21.15.90:0/4067064233 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:07.357 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597b7fe640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f597c100750 0x7f597c19bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 -- 172.21.15.90:0/4067064233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f597c100750 msgr2=0x7f597c19bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f597c100750 0x7f597c19bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 -- 172.21.15.90:0/4067064233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970009560 con 0x7f597c0ffda0 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.336+0000 7f597bfff640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 0x7f597c19b800 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f596c00b420 tx=0x7f596c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.337+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596c011800 con 0x7f597c0ffda0 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.337+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f596c011df0 con 0x7f597c0ffda0 2024-09-13T15:32:07.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.337+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597c195bd0 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.337+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f596c010e80 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.337+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597c196120 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.337+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f597c10cf60 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.338+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f596c011960 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.338+0000 7f59797fa640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f596406cc50 0x7f596406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f596c08bc00 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f597b7fe640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f596406cc50 0x7f596406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f59797fa640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] conn(0x7f5964072fb0 0x7f59640753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 --> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5964075a80 con 0x7f5964072fb0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f596c0912e0 con 0x7f597c0ffda0 2024-09-13T15:32:07.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f59809c0640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] conn(0x7f5964072fb0 0x7f59640753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f597b7fe640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f596406cc50 0x7f596406f110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f597c196d60 tx=0x7f5970005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.339+0000 7f59809c0640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] conn(0x7f5964072fb0 0x7f59640753d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.340+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== osd.0 v2:172.21.15.90:6802/679705213 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f5964075a80 con 0x7f5964072fb0 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 --> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f597c196660 con 0x7f5964072fb0 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f59797fa640 1 -- 172.21.15.90:0/4067064233 <== osd.0 v2:172.21.15.90:6802/679705213 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f597c196660 con 0x7f5964072fb0 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] conn(0x7f5964072fb0 msgr2=0x7f59640753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] conn(0x7f5964072fb0 0x7f59640753d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f596406cc50 msgr2=0x7f596406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f596406cc50 0x7f596406f110 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f597c196d60 tx=0x7f5970005a20 comp rx=0 tx=0).stop 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 msgr2=0x7f597c19b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 0x7f597c19b800 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f596c00b420 tx=0x7f596c00b8e0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.360 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 shutdown_connections 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f596406cc50 0x7f596406f110 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:6802/679705213,v1:172.21.15.90:6803/679705213] conn(0x7f5964072fb0 0x7f59640753d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f597c100750 0x7f597c19bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 --2- 172.21.15.90:0/4067064233 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f597c0ffda0 0x7f597c19b800 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 >> 172.21.15.90:0/4067064233 conn(0x7f597c0fb840 msgr2=0x7f597c0fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 shutdown_connections 2024-09-13T15:32:07.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.353+0000 7f5982423640 1 -- 172.21.15.90:0/4067064233 wait complete. 2024-09-13T15:32:07.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 -- 172.21.15.90:0/3966853741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e34100750 msgr2=0x7f3e341047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 --2- 172.21.15.90:0/3966853741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e34100750 0x7f3e341047a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3e28009880 tx=0x7f3e2802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 -- 172.21.15.90:0/3966853741 shutdown_connections 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 --2- 172.21.15.90:0/3966853741 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e34100750 0x7f3e341047a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 --2- 172.21.15.90:0/3966853741 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e340ffda0 0x7f3e34100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 -- 172.21.15.90:0/3966853741 >> 172.21.15.90:0/3966853741 conn(0x7f3e340fb840 msgr2=0x7f3e340fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 -- 172.21.15.90:0/3966853741 shutdown_connections 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 -- 172.21.15.90:0/3966853741 wait complete. 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.744+0000 7f3e39084640 1 Processor -- start 2024-09-13T15:32:07.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e39084640 1 -- start start 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e39084640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 0x7f3e340727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e39084640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e34100750 0x7f3e34072cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e39084640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e34074780 con 0x7f3e340ffda0 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e39084640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e34073230 con 0x7f3e34100750 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 0x7f3e340727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 0x7f3e340727b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51472/0 (socket says 172.21.15.90:51472) 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 -- 172.21.15.90:0/1722414864 learned_addr learned my addr 172.21.15.90:0/1722414864 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e3259c640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e34100750 0x7f3e34072cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 -- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e34100750 msgr2=0x7f3e34072cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e34100750 0x7f3e34072cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 -- 172.21.15.90:0/1722414864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28009560 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.745+0000 7f3e32d9d640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 0x7f3e340727b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c8a0 tx=0x7f3e2400cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.746+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e24007b80 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.746+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e34073510 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.746+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3e24007ce0 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.746+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e24018630 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.746+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e341a1e30 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.746+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e3410cee0 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.747+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3e2400f450 con 0x7f3e340ffda0 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.747+0000 7f3e1bfff640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3e1c06cc50 0x7f3e1c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f3e2408c570 con 0x7f3e340ffda0 2024-09-13T15:32:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e3259c640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3e1c06cc50 0x7f3e1c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e1bfff640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] conn(0x7f3e1c072fb0 0x7f3e1c0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 --> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e1c075a80 con 0x7f3e1c072fb0 2024-09-13T15:32:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e2408ec40 con 0x7f3e340ffda0 2024-09-13T15:32:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e3259c640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3e1c06cc50 0x7f3e1c06f110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e28006010 tx=0x7f3e28000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:07.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e3359e640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] conn(0x7f3e1c072fb0 0x7f3e1c0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.748+0000 7f3e3359e640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] conn(0x7f3e1c072fb0 0x7f3e1c0753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:07.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.749+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== osd.6 v2:172.21.15.175:6816/1206614294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f3e1c075a80 con 0x7f3e1c072fb0 2024-09-13T15:32:07.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.762+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 --> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e34101280 con 0x7f3e1c072fb0 2024-09-13T15:32:07.764 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e1bfff640 1 -- 172.21.15.90:0/1722414864 <== osd.6 v2:172.21.15.175:6816/1206614294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3e34101280 con 0x7f3e1c072fb0 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] conn(0x7f3e1c072fb0 msgr2=0x7f3e1c0753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] conn(0x7f3e1c072fb0 0x7f3e1c0753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3e1c06cc50 msgr2=0x7f3e1c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3e1c06cc50 0x7f3e1c06f110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3e28006010 tx=0x7f3e28000bc0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 msgr2=0x7f3e340727b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 0x7f3e340727b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c8a0 tx=0x7f3e2400cd60 comp rx=0 tx=0).stop 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 shutdown_connections 2024-09-13T15:32:07.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:6816/1206614294,v1:172.21.15.175:6817/1206614294] conn(0x7f3e1c072fb0 0x7f3e1c0753d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3e1c06cc50 0x7f3e1c06f110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e34100750 0x7f3e34072cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 --2- 172.21.15.90:0/1722414864 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3e340ffda0 0x7f3e340727b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 >> 172.21.15.90:0/1722414864 conn(0x7f3e340fb840 msgr2=0x7f3e340fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 shutdown_connections 2024-09-13T15:32:07.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:07.763+0000 7f3e39084640 1 -- 172.21.15.90:0/1722414864 wait complete. 2024-09-13T15:32:07.990 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:07 smithi090 ceph-mon[27811]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:07 smithi175 ceph-mon[36128]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:08.571 INFO:teuthology.orchestra.run.smithi090.stdout:81604378658 2024-09-13T15:32:08.572 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 -- 172.21.15.90:0/625519604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e00fe9f0 msgr2=0x7fb3e00fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 --2- 172.21.15.90:0/625519604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e00fe9f0 0x7fb3e00fee50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0009880 tx=0x7fb3d002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 -- 172.21.15.90:0/625519604 shutdown_connections 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 --2- 172.21.15.90:0/625519604 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e00fe9f0 0x7fb3e00fee50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 --2- 172.21.15.90:0/625519604 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3e01049f0 0x7fb3e0104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 -- 172.21.15.90:0/625519604 >> 172.21.15.90:0/625519604 conn(0x7fb3e00758a0 msgr2=0x7fb3e00fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 -- 172.21.15.90:0/625519604 shutdown_connections 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.736+0000 7fb3e7364640 1 -- 172.21.15.90:0/625519604 wait complete. 2024-09-13T15:32:08.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e7364640 1 Processor -- start 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e7364640 1 -- start start 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e7364640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3e00fe9f0 0x7fb3e01968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e7364640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 0x7fb3e0196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e7364640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0197320 con 0x7fb3e01049f0 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e7364640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0197490 con 0x7fb3e00fe9f0 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 0x7fb3e0196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 0x7fb3e0196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51486/0 (socket says 172.21.15.90:51486) 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 -- 172.21.15.90:0/256155501 learned_addr learned my addr 172.21.15.90:0/256155501 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 -- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3e00fe9f0 msgr2=0x7fb3e01968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3e00fe9f0 0x7fb3e01968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 -- 172.21.15.90:0/256155501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0009560 con 0x7fb3e01049f0 2024-09-13T15:32:08.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.737+0000 7fb3e48ff640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 0x7fb3e0196de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0006010 tx=0x7fb3d0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.738+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d0005460 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.738+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb3d00055c0 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.738+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e0197710 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.738+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3d003c900 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.738+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e019fc40 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.738+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3e010be10 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.739+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb3d0003c10 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.739+0000 7fb3d67fc640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb3cc06cc50 0x7fb3cc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3e5100640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb3cc06cc50 0x7fb3cc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fb3d00ad2f0 con 0x7fb3e01049f0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3d67fc640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] conn(0x7fb3cc072fb0 0x7fb3cc0753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 --> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3cc075a80 con 0x7fb3cc072fb0 2024-09-13T15:32:08.754 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3d00af9c0 con 0x7fb3e01049f0 2024-09-13T15:32:08.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3e5100640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb3cc06cc50 0x7fb3cc06f110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009710 tx=0x7fb3dc006c90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:08.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.740+0000 7fb3e5901640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] conn(0x7fb3cc072fb0 0x7fb3cc0753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:08.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.741+0000 7fb3e5901640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] conn(0x7fb3cc072fb0 0x7fb3cc0753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:08.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.741+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== osd.7 v2:172.21.15.175:6824/1677596558 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb3cc075a80 con 0x7fb3cc072fb0 2024-09-13T15:32:08.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.754+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 --> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb3e0060b30 con 0x7fb3cc072fb0 2024-09-13T15:32:08.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3d67fc640 1 -- 172.21.15.90:0/256155501 <== osd.7 v2:172.21.15.175:6824/1677596558 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb3e0060b30 con 0x7fb3cc072fb0 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] conn(0x7fb3cc072fb0 msgr2=0x7fb3cc0753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] conn(0x7fb3cc072fb0 0x7fb3cc0753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb3cc06cc50 msgr2=0x7fb3cc06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb3cc06cc50 0x7fb3cc06f110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009710 tx=0x7fb3dc006c90 comp rx=0 tx=0).stop 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 msgr2=0x7fb3e0196de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.755+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 0x7fb3e0196de0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0006010 tx=0x7fb3d0004610 comp rx=0 tx=0).stop 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 shutdown_connections 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:6824/1677596558,v1:172.21.15.175:6825/1677596558] conn(0x7fb3cc072fb0 0x7fb3cc0753d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb3cc06cc50 0x7fb3cc06f110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb3e01049f0 0x7fb3e0196de0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.759 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 --2- 172.21.15.90:0/256155501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3e00fe9f0 0x7fb3e01968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:08.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 >> 172.21.15.90:0/256155501 conn(0x7fb3e00758a0 msgr2=0x7fb3e0102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:08.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 shutdown_connections 2024-09-13T15:32:08.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:08.756+0000 7fb3e7364640 1 -- 172.21.15.90:0/256155501 wait complete. 2024-09-13T15:32:09.393 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 -- 172.21.15.90:0/2644377310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0fe9f0 msgr2=0x7feb9c0fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 --2- 172.21.15.90:0/2644377310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0fe9f0 0x7feb9c0fee50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feb90009880 tx=0x7feb9002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 -- 172.21.15.90:0/2644377310 shutdown_connections 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 --2- 172.21.15.90:0/2644377310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c0fe9f0 0x7feb9c0fee50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 --2- 172.21.15.90:0/2644377310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb9c1049f0 0x7feb9c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 -- 172.21.15.90:0/2644377310 >> 172.21.15.90:0/2644377310 conn(0x7feb9c0758a0 msgr2=0x7feb9c0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 -- 172.21.15.90:0/2644377310 shutdown_connections 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.369+0000 7feba1464640 1 -- 172.21.15.90:0/2644377310 wait complete. 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.370+0000 7feba1464640 1 Processor -- start 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.370+0000 7feba1464640 1 -- start start 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.370+0000 7feba1464640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb9c0fe9f0 0x7feb9c1968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.370+0000 7feba1464640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 0x7feb9c196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:09.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.370+0000 7feba1464640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c197360 con 0x7feb9c1049f0 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.370+0000 7feba1464640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c1974d0 con 0x7feb9c0fe9f0 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 0x7feb9c196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 0x7feb9c196e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:51500/0 (socket says 172.21.15.90:51500) 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 -- 172.21.15.90:0/1609132597 learned_addr learned my addr 172.21.15.90:0/1609132597 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9affd640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb9c0fe9f0 0x7feb9c1968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 -- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb9c0fe9f0 msgr2=0x7feb9c1968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:09.395 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb9c0fe9f0 0x7feb9c1968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 -- 172.21.15.90:0/1609132597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb90009560 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb9a7fc640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 0x7feb9c196e20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feb90006010 tx=0x7feb90004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb90005460 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7feb900055c0 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c197750 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb9003c900 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.371+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c19fc30 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.372+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb9c10be10 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7feb90003c10 con 0x7feb9c1049f0 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb83fff640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7feb8406cc50 0x7feb8406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:09.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7feb900ad3f0 con 0x7feb9c1049f0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb9affd640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7feb8406cc50 0x7feb8406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb83fff640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] conn(0x7feb84072fb0 0x7feb840753d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 --> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feb84075a80 con 0x7feb84072fb0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.373+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb900afac0 con 0x7feb9c1049f0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.374+0000 7feb9b7fe640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] conn(0x7feb84072fb0 0x7feb840753d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.374+0000 7feb9affd640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7feb8406cc50 0x7feb8406f110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009890 tx=0x7feb8c008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.374+0000 7feb9b7fe640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] conn(0x7feb84072fb0 0x7feb840753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.375+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== osd.4 v2:172.21.15.175:6800/2031407459 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7feb84075a80 con 0x7feb84072fb0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 --> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb9c197750 con 0x7feb84072fb0 2024-09-13T15:32:09.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feb83fff640 1 -- 172.21.15.90:0/1609132597 <== osd.4 v2:172.21.15.175:6800/2031407459 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feb9c197750 con 0x7feb84072fb0 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] conn(0x7feb84072fb0 msgr2=0x7feb840753d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] conn(0x7feb84072fb0 0x7feb840753d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7feb8406cc50 msgr2=0x7feb8406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7feb8406cc50 0x7feb8406f110 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009890 tx=0x7feb8c008040 comp rx=0 tx=0).stop 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 msgr2=0x7feb9c196e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 0x7feb9c196e20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feb90006010 tx=0x7feb90004610 comp rx=0 tx=0).stop 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 shutdown_connections 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:6800/2031407459,v1:172.21.15.175:6801/2031407459] conn(0x7feb84072fb0 0x7feb840753d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7feb8406cc50 0x7feb8406f110 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.398 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7feb9c1049f0 0x7feb9c196e20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 --2- 172.21.15.90:0/1609132597 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb9c0fe9f0 0x7feb9c1968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.388+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 >> 172.21.15.90:0/1609132597 conn(0x7feb9c0758a0 msgr2=0x7feb9c102380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.389+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 shutdown_connections 2024-09-13T15:32:09.399 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:09.389+0000 7feba1464640 1 -- 172.21.15.90:0/1609132597 wait complete. 2024-09-13T15:32:10.197 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:09 smithi090 ceph-mon[27811]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:10.220 INFO:teuthology.orchestra.run.smithi090.stdout:201863462927 2024-09-13T15:32:10.220 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-13T15:32:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:09 smithi175 ceph-mon[36128]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:10.638 INFO:teuthology.orchestra.run.smithi090.stdout:236223201290 2024-09-13T15:32:10.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-13T15:32:11.024 INFO:teuthology.orchestra.run.smithi090.stdout:38654705708 2024-09-13T15:32:11.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-13T15:32:11.499 INFO:teuthology.orchestra.run.smithi090.stdout:133143986201 2024-09-13T15:32:11.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-13T15:32:11.700 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:11.703 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:11.704 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:11.705 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:11.711 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:11.712 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:12.190 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:11 smithi090 ceph-mon[27811]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:11 smithi175 ceph-mon[36128]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:12.401 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:14.107 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:13 smithi090 ceph-mon[27811]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:13 smithi175 ceph-mon[36128]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:16.103 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:15 smithi090 ceph-mon[27811]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:15 smithi175 ceph-mon[36128]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:17.942 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:17 smithi090 ceph-mon[27811]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:17 smithi175 ceph-mon[36128]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:18.939 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:20.060 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:19 smithi090 ceph-mon[27811]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:20.110 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:19 smithi175 ceph-mon[36128]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:20.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 -- 172.21.15.90:0/1941948453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 msgr2=0x7f5948104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 --2- 172.21.15.90:0/1941948453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f5948104dd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5938009880 tx=0x7f593802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 -- 172.21.15.90:0/1941948453 shutdown_connections 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 --2- 172.21.15.90:0/1941948453 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 0x7f59480fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 --2- 172.21.15.90:0/1941948453 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f5948104dd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 -- 172.21.15.90:0/1941948453 >> 172.21.15.90:0/1941948453 conn(0x7f59480758a0 msgr2=0x7f59480fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 -- 172.21.15.90:0/1941948453 shutdown_connections 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.786+0000 7f594de83640 1 -- 172.21.15.90:0/1941948453 wait complete. 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f594de83640 1 Processor -- start 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f594de83640 1 -- start start 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f594de83640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 0x7f594819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f594de83640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f594819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f594de83640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594819b080 con 0x7f59481049f0 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f594de83640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59481946a0 con 0x7f59480fe9f0 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f5946ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f594819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f5946ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f594819aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48570/0 (socket says 172.21.15.90:48570) 2024-09-13T15:32:20.424 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.787+0000 7f5946ffd640 1 -- 172.21.15.90:0/2680908651 learned_addr learned my addr 172.21.15.90:0/2680908651 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:20.425 INFO:teuthology.orchestra.run.smithi090.stdout:167503724566 2024-09-13T15:32:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f59477fe640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 0x7f594819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5946ffd640 1 -- 172.21.15.90:0/2680908651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 msgr2=0x7f594819a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5946ffd640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 0x7f594819a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5946ffd640 1 -- 172.21.15.90:0/2680908651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5938009560 con 0x7f59481049f0 2024-09-13T15:32:20.425 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f59477fe640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 0x7f594819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5946ffd640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f594819aaf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f593c007b50 tx=0x7f593c00e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f593c011070 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f593c00c2e0 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5948194920 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f593c00ad10 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.788+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5948194e70 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.789+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5948100130 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.790+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f593c00eab0 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.790+0000 7f5944ff9640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f593006cc60 0x7f593006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.790+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f593c08dac0 con 0x7f59481049f0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.790+0000 7f59477fe640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f593006cc60 0x7f593006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.790+0000 7f59477fe640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f593006cc60 0x7f593006f120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5938005ab0 tx=0x7f5938005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:20.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.794+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f593c057e60 con 0x7f59481049f0 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.977+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f594810bab0 con 0x7f59481049f0 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.977+0000 7f5944ff9640 1 -- 172.21.15.90:0/2680908651 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f593c05b480 con 0x7f59481049f0 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f593006cc60 msgr2=0x7f593006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f593006cc60 0x7f593006f120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5938005ab0 tx=0x7f5938005a20 comp rx=0 tx=0).stop 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 msgr2=0x7f594819aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f594819aaf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f593c007b50 tx=0x7f593c00e420 comp rx=0 tx=0).stop 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 shutdown_connections 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f593006cc60 0x7f593006f120 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f59481049f0 0x7f594819aaf0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 --2- 172.21.15.90:0/2680908651 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59480fe9f0 0x7f594819a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 >> 172.21.15.90:0/2680908651 conn(0x7f59480758a0 msgr2=0x7f59480fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 shutdown_connections 2024-09-13T15:32:20.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:19.979+0000 7f594de83640 1 -- 172.21.15.90:0/2680908651 wait complete. 2024-09-13T15:32:20.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 -- 172.21.15.90:0/4172321997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 msgr2=0x7f3714104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 --2- 172.21.15.90:0/4172321997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714104dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3704009880 tx=0x7f370402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 -- 172.21.15.90:0/4172321997 shutdown_connections 2024-09-13T15:32:20.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 --2- 172.21.15.90:0/4172321997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37140fe9f0 0x7f37140fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 --2- 172.21.15.90:0/4172321997 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714104dd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 -- 172.21.15.90:0/4172321997 >> 172.21.15.90:0/4172321997 conn(0x7f37140758a0 msgr2=0x7f37140fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 -- 172.21.15.90:0/4172321997 shutdown_connections 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 -- 172.21.15.90:0/4172321997 wait complete. 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 Processor -- start 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.803+0000 7f3719e0c640 1 -- start start 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f3719e0c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37140fe9f0 0x7f3714196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f3719e0c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f3719e0c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714197300 con 0x7f37141049f0 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f3719e0c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714197470 con 0x7f37140fe9f0 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48592/0 (socket says 172.21.15.90:48592) 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 -- 172.21.15.90:0/3243837675 learned_addr learned my addr 172.21.15.90:0/3243837675 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f37137fe640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37140fe9f0 0x7f3714196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:20.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 -- 172.21.15.90:0/3243837675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37140fe9f0 msgr2=0x7f3714196880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37140fe9f0 0x7f3714196880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 -- 172.21.15.90:0/3243837675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704009560 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f370affd640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714196dc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00b420 tx=0x7f36fc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36fc011800 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371419f8f0 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f36fc011df0 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.804+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f36fc010e80 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.805+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371419fca0 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.806+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3714100130 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.806+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f36fc011960 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.806+0000 7f37117fa640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f370006cc50 0x7f370006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.807+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f36fc08bbf0 con 0x7f37141049f0 2024-09-13T15:32:20.811 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.807+0000 7f37137fe640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f370006cc50 0x7f370006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:20.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.807+0000 7f37137fe640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f370006cc50 0x7f370006f110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3704009230 tx=0x7f37040058d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:20.813 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.811+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f36fc055f10 con 0x7f37141049f0 2024-09-13T15:32:20.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.980+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f37140feed0 con 0x7f37141049f0 2024-09-13T15:32:20.982 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:20 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2680908651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T15:32:20.983 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.980+0000 7f37117fa640 1 -- 172.21.15.90:0/3243837675 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f36fc059530 con 0x7f37141049f0 2024-09-13T15:32:20.983 INFO:teuthology.orchestra.run.smithi090.stdout:38654705711 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.982+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f370006cc50 msgr2=0x7f370006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.982+0000 7f3719e0c640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f370006cc50 0x7f370006f110 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3704009230 tx=0x7f37040058d0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.982+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 msgr2=0x7f3714196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.982+0000 7f3719e0c640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714196dc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36fc00b420 tx=0x7f36fc00b8e0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 shutdown_connections 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f370006cc50 0x7f370006f110 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f37141049f0 0x7f3714196dc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 --2- 172.21.15.90:0/3243837675 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37140fe9f0 0x7f3714196880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 >> 172.21.15.90:0/3243837675 conn(0x7f37140758a0 msgr2=0x7f37140fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:20.985 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 shutdown_connections 2024-09-13T15:32:20.986 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:20.983+0000 7f3719e0c640 1 -- 172.21.15.90:0/3243837675 wait complete. 2024-09-13T15:32:21.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:20 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2680908651' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-13T15:32:21.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 -- 172.21.15.90:0/2675534571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 msgr2=0x7f256c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 --2- 172.21.15.90:0/2675534571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c100180 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f255c009880 tx=0x7f255c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 -- 172.21.15.90:0/2675534571 shutdown_connections 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 --2- 172.21.15.90:0/2675534571 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f256c100750 0x7f256c1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 --2- 172.21.15.90:0/2675534571 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c100180 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 -- 172.21.15.90:0/2675534571 >> 172.21.15.90:0/2675534571 conn(0x7f256c0fb840 msgr2=0x7f256c0fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 -- 172.21.15.90:0/2675534571 shutdown_connections 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.679+0000 7f2571d69640 1 -- 172.21.15.90:0/2675534571 wait complete. 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f2571d69640 1 Processor -- start 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f2571d69640 1 -- start start 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f2571d69640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f2571d69640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f256c100750 0x7f256c195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:21.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f2571d69640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c196590 con 0x7f256c0ffda0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f2571d69640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f256c196700 con 0x7f256c100750 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f256b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f256b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c195a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48622/0 (socket says 172.21.15.90:48622) 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.680+0000 7f256b7fe640 1 -- 172.21.15.90:0/542768891 learned_addr learned my addr 172.21.15.90:0/542768891 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f256affd640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f256c100750 0x7f256c195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f256b7fe640 1 -- 172.21.15.90:0/542768891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f256c100750 msgr2=0x7f256c195f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f256b7fe640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f256c100750 0x7f256c195f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f256b7fe640 1 -- 172.21.15.90:0/542768891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c009560 con 0x7f256c0ffda0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f256b7fe640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c195a00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f255c006010 tx=0x7f255c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f255c005460 con 0x7f256c0ffda0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f255c0055c0 con 0x7f256c0ffda0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f255c03c900 con 0x7f256c0ffda0 2024-09-13T15:32:21.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f256c19a460 con 0x7f256c0ffda0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.681+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f256c19a950 con 0x7f256c0ffda0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.682+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256c101280 con 0x7f256c0ffda0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.682+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f255c003c10 con 0x7f256c0ffda0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.683+0000 7f2568ff9640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f255406cc50 0x7f255406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.683+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f255c0ad550 con 0x7f256c0ffda0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.683+0000 7f256affd640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f255406cc50 0x7f255406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.683+0000 7f256affd640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f255406cc50 0x7f255406f110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f256c196fb0 tx=0x7f2560005cc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:21.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.687+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f255c077870 con 0x7f256c0ffda0 2024-09-13T15:32:21.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.852+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f256c196d60 con 0x7f256c0ffda0 2024-09-13T15:32:21.854 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.852+0000 7f2568ff9640 1 -- 172.21.15.90:0/542768891 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f255c07ae90 con 0x7f256c0ffda0 2024-09-13T15:32:21.855 INFO:teuthology.orchestra.run.smithi090.stdout:111669149727 2024-09-13T15:32:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f255406cc50 msgr2=0x7f255406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:21.856 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f255406cc50 0x7f255406f110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f256c196fb0 tx=0x7f2560005cc0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 msgr2=0x7f256c195a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c195a00 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f255c006010 tx=0x7f255c004610 comp rx=0 tx=0).stop 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 shutdown_connections 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f255406cc50 0x7f255406f110 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f256c100750 0x7f256c195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 --2- 172.21.15.90:0/542768891 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f256c0ffda0 0x7f256c195a00 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 >> 172.21.15.90:0/542768891 conn(0x7f256c0fb840 msgr2=0x7f256c0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 shutdown_connections 2024-09-13T15:32:21.857 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:21.854+0000 7f2571d69640 1 -- 172.21.15.90:0/542768891 wait complete. 2024-09-13T15:32:22.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:21 smithi090 ceph-mon[27811]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:22.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:21 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3243837675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T15:32:22.221 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:21 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/542768891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T15:32:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:21 smithi175 ceph-mon[36128]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:21 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3243837675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-13T15:32:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:21 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/542768891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-13T15:32:22.489 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724563 got 167503724566 for osd.5 2024-09-13T15:32:22.489 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1471305213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec4104b60 msgr2=0x7f2ec4104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1471305213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec4104b60 0x7f2ec4104f40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009880 tx=0x7f2ec002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1471305213 shutdown_connections 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1471305213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40fe3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1471305213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec4104b60 0x7f2ec4104f40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1471305213 >> 172.21.15.90:0/1471305213 conn(0x7f2ec40f9e90 msgr2=0x7f2ec40fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1471305213 shutdown_connections 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.646+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1471305213 wait complete. 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ecbff7640 1 Processor -- start 2024-09-13T15:32:22.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ecbff7640 1 -- start start 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ecbff7640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40ff160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ecbff7640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec4104b60 0x7f2ec40ff6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ecbff7640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec4101130 con 0x7f2ec40fdf80 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ecbff7640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec40ffbe0 con 0x7f2ec4104b60 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9d93640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40ff160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9d93640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40ff160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:48630/0 (socket says 172.21.15.90:48630) 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9d93640 1 -- 172.21.15.90:0/1713026379 learned_addr learned my addr 172.21.15.90:0/1713026379 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9592640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec4104b60 0x7f2ec40ff6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9d93640 1 -- 172.21.15.90:0/1713026379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec4104b60 msgr2=0x7f2ec40ff6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9d93640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec4104b60 0x7f2ec40ff6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.647+0000 7f2ec9d93640 1 -- 172.21.15.90:0/1713026379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0009560 con 0x7f2ec40fdf80 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.648+0000 7f2ec9d93640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40ff160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0006010 tx=0x7f2ec0004d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:22.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.648+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ec00054a0 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.648+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec40ffe60 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.648+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2ec0005600 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.648+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ec003c800 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.648+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec41a1e40 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.649+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec41012a0 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.649+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f2ec0003b80 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.650+0000 7f2ebaffd640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f2eb006cc50 0x7f2eb006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.650+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f2ec00ad370 con 0x7f2ec40fdf80 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.650+0000 7f2ec9592640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f2eb006cc50 0x7f2eb006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.650+0000 7f2ec9592640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f2eb006cc50 0x7f2eb006f110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ec41008e0 tx=0x7f2eb400b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:22.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.654+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f2ec00777c0 con 0x7f2ec40fdf80 2024-09-13T15:32:22.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.820+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2ec41080a0 con 0x7f2ec40fdf80 2024-09-13T15:32:22.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.820+0000 7f2ebaffd640 1 -- 172.21.15.90:0/1713026379 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ec007ade0 con 0x7f2ec40fdf80 2024-09-13T15:32:22.823 INFO:teuthology.orchestra.run.smithi090.stdout:81604378661 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f2eb006cc50 msgr2=0x7f2eb006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f2eb006cc50 0x7f2eb006f110 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2ec41008e0 tx=0x7f2eb400b390 comp rx=0 tx=0).stop 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 msgr2=0x7f2ec40ff160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40ff160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0006010 tx=0x7f2ec0004d80 comp rx=0 tx=0).stop 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 shutdown_connections 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f2eb006cc50 0x7f2eb006f110 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec4104b60 0x7f2ec40ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.822+0000 7f2ecbff7640 1 --2- 172.21.15.90:0/1713026379 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2ec40fdf80 0x7f2ec40ff160 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:22.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.823+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 >> 172.21.15.90:0/1713026379 conn(0x7f2ec40f9e90 msgr2=0x7f2ec40fb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.823+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 shutdown_connections 2024-09-13T15:32:22.826 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:22.823+0000 7f2ecbff7640 1 -- 172.21.15.90:0/1713026379 wait complete. 2024-09-13T15:32:23.184 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:22 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1713026379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T15:32:23.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:22 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1713026379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-13T15:32:23.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378658 got 81604378661 for osd.2 2024-09-13T15:32:24.207 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:23 smithi090 ceph-mon[27811]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:23 smithi175 ceph-mon[36128]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:24.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149724 got 111669149727 for osd.3 2024-09-13T15:32:24.363 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:24.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.529+0000 7f07c4540640 1 -- 172.21.15.90:0/581348137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 msgr2=0x7f07bc0fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.529+0000 7f07c4540640 1 --2- 172.21.15.90:0/581348137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc0fee50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009880 tx=0x7f07b802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- 172.21.15.90:0/581348137 shutdown_connections 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 --2- 172.21.15.90:0/581348137 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc0fee50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 --2- 172.21.15.90:0/581348137 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07bc1049f0 0x7f07bc104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- 172.21.15.90:0/581348137 >> 172.21.15.90:0/581348137 conn(0x7f07bc0758a0 msgr2=0x7f07bc0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- 172.21.15.90:0/581348137 shutdown_connections 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- 172.21.15.90:0/581348137 wait complete. 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 Processor -- start 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- start start 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc196860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07bc1049f0 0x7f07bc196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc197330 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.530+0000 7f07c4540640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07bc197470 con 0x7f07bc1049f0 2024-09-13T15:32:24.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc196860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc196860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57076/0 (socket says 172.21.15.90:57076) 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 -- 172.21.15.90:0/2359121063 learned_addr learned my addr 172.21.15.90:0/2359121063 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c1adb640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07bc1049f0 0x7f07bc196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 -- 172.21.15.90:0/2359121063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07bc1049f0 msgr2=0x7f07bc196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07bc1049f0 0x7f07bc196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 -- 172.21.15.90:0/2359121063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b8009560 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c22dc640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc196860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00c8f0 tx=0x7f07ac00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ac007b80 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f07ac007ce0 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ac004e10 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07bc19f8d0 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.531+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07bc19fc80 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.532+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07bc100130 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.533+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f07ac005020 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.533+0000 7f07b37fe640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f07a806cc50 0x7f07a806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:24.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.533+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f07ac08c0d0 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.533+0000 7f07c1adb640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f07a806cc50 0x7f07a806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:24.537 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.534+0000 7f07c1adb640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f07a806cc50 0x7f07a806f110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009230 tx=0x7f07b80058d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:24.539 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.537+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f07ac0564a0 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.708+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f07bc0fee50 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.709+0000 7f07b37fe640 1 -- 172.21.15.90:0/2359121063 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07ac059ac0 con 0x7f07bc0fe9f0 2024-09-13T15:32:24.711 INFO:teuthology.orchestra.run.smithi090.stdout:60129542187 2024-09-13T15:32:24.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f07a806cc50 msgr2=0x7f07a806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f07a806cc50 0x7f07a806f110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009230 tx=0x7f07b80058d0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 msgr2=0x7f07bc196860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc196860 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f07ac00c8f0 tx=0x7f07ac00cdb0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 shutdown_connections 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f07a806cc50 0x7f07a806f110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07bc1049f0 0x7f07bc196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 --2- 172.21.15.90:0/2359121063 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f07bc0fe9f0 0x7f07bc196860 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 >> 172.21.15.90:0/2359121063 conn(0x7f07bc0758a0 msgr2=0x7f07bc102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 shutdown_connections 2024-09-13T15:32:24.715 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.712+0000 7f07c4540640 1 -- 172.21.15.90:0/2359121063 wait complete. 2024-09-13T15:32:24.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 -- 172.21.15.90:0/496096300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 msgr2=0x7fe0e4101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/496096300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e4101130 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc009880 tx=0x7fe0cc02a530 comp rx=0 tx=0).stop 2024-09-13T15:32:24.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 -- 172.21.15.90:0/496096300 shutdown_connections 2024-09-13T15:32:24.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/496096300 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 0x7fe0e4109b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/496096300 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e4101130 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.795 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 -- 172.21.15.90:0/496096300 >> 172.21.15.90:0/496096300 conn(0x7fe0e40758a0 msgr2=0x7fe0e40fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 -- 172.21.15.90:0/496096300 shutdown_connections 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.756+0000 7fe0e8a23640 1 -- 172.21.15.90:0/496096300 wait complete. 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.757+0000 7fe0e8a23640 1 Processor -- start 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.757+0000 7fe0e8a23640 1 -- start start 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e8a23640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e419b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e8a23640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 0x7fe0e419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e8a23640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e419c200 con 0x7fe0e4100d50 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e8a23640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4195760 con 0x7fe0e4101670 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e419b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e419b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57092/0 (socket says 172.21.15.90:57092) 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 -- 172.21.15.90:0/913054330 learned_addr learned my addr 172.21.15.90:0/913054330 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:24.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e1d9b640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 0x7fe0e419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 -- 172.21.15.90:0/913054330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 msgr2=0x7fe0e419bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 0x7fe0e419bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 -- 172.21.15.90:0/913054330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0cc009560 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e1d9b640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 0x7fe0e419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e259c640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e419b670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005940 tx=0x7fe0cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0cc004c90 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e41959e0 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe0cc003a80 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0cc03cab0 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.758+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e4195ea0 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.759+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e4068470 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.760+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe0cc02ad50 con 0x7fe0e4100d50 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.760+0000 7fe0db7fe640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe0d006cc50 0x7fe0d006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:24.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.761+0000 7fe0e1d9b640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe0d006cc50 0x7fe0d006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:24.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.761+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fe0cc0ad340 con 0x7fe0e4100d50 2024-09-13T15:32:24.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.761+0000 7fe0e1d9b640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe0d006cc50 0x7fe0d006f110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4196bd0 tx=0x7fe0d4005d30 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:24.798 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.765+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fe0cc077660 con 0x7fe0e4100d50 2024-09-13T15:32:24.932 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.930+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe0e4196d10 con 0x7fe0e4100d50 2024-09-13T15:32:24.933 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:24 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2359121063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T15:32:24.933 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.930+0000 7fe0db7fe640 1 -- 172.21.15.90:0/913054330 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe0cc07ac80 con 0x7fe0e4100d50 2024-09-13T15:32:24.933 INFO:teuthology.orchestra.run.smithi090.stdout:133143986204 2024-09-13T15:32:24.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe0d006cc50 msgr2=0x7fe0d006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe0d006cc50 0x7fe0d006f110 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4196bd0 tx=0x7fe0d4005d30 comp rx=0 tx=0).stop 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 msgr2=0x7fe0e419b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e419b670 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc005940 tx=0x7fe0cc0046c0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 shutdown_connections 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe0d006cc50 0x7fe0d006f110 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe0e4101670 0x7fe0e419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 --2- 172.21.15.90:0/913054330 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe0e4100d50 0x7fe0e419b670 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 >> 172.21.15.90:0/913054330 conn(0x7fe0e40758a0 msgr2=0x7fe0e4101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 shutdown_connections 2024-09-13T15:32:24.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:24.932+0000 7fe0e8a23640 1 -- 172.21.15.90:0/913054330 wait complete. 2024-09-13T15:32:25.186 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:24 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2359121063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-13T15:32:25.623 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542187 for osd.1 2024-09-13T15:32:25.623 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:26.085 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705708 got 38654705711 for osd.0 2024-09-13T15:32:26.085 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:26.172 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:25 smithi090 ceph-mon[27811]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:26.173 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:25 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/913054330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T15:32:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:25 smithi175 ceph-mon[36128]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:25 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/913054330' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-13T15:32:26.695 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986201 got 133143986204 for osd.4 2024-09-13T15:32:26.695 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 -- 172.21.15.90:0/2916662700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474101630 msgr2=0x7f8474109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 --2- 172.21.15.90:0/2916662700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474101630 0x7f8474109b20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84640098b0 tx=0x7f846402a560 comp rx=0 tx=0).stop 2024-09-13T15:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 -- 172.21.15.90:0/2916662700 shutdown_connections 2024-09-13T15:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 --2- 172.21.15.90:0/2916662700 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474101630 0x7f8474109b20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 --2- 172.21.15.90:0/2916662700 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8474100d10 0x7f84741010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 -- 172.21.15.90:0/2916662700 >> 172.21.15.90:0/2916662700 conn(0x7f84740758a0 msgr2=0x7f84740fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 -- 172.21.15.90:0/2916662700 shutdown_connections 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.130+0000 7f847c737640 1 -- 172.21.15.90:0/2916662700 wait complete. 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847c737640 1 Processor -- start 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847c737640 1 -- start start 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847c737640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 0x7f847419b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847c737640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8474101630 0x7f847419bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847c737640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847419c2a0 con 0x7f8474100d10 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847c737640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84741958c0 con 0x7f8474101630 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847a4d3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 0x7f847419b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847a4d3640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 0x7f847419b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57110/0 (socket says 172.21.15.90:57110) 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847a4d3640 1 -- 172.21.15.90:0/3557360133 learned_addr learned my addr 172.21.15.90:0/3557360133 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:27.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f8479cd2640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8474101630 0x7f847419bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847a4d3640 1 -- 172.21.15.90:0/3557360133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8474101630 msgr2=0x7f847419bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847a4d3640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8474101630 0x7f847419bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.131+0000 7f847a4d3640 1 -- 172.21.15.90:0/3557360133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464009560 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.132+0000 7f847a4d3640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 0x7f847419b7d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f847000b420 tx=0x7f847000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.132+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8470011800 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.132+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8470011df0 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.132+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8474195ba0 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.132+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8470010e80 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.132+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84741960c0 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.133+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84740684b0 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.133+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8470011960 con 0x7f8474100d10 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.134+0000 7f846b7fe640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8460071070 0x7f8460073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:27.140 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.134+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f847008bbe0 con 0x7f8474100d10 2024-09-13T15:32:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.134+0000 7f8479cd2640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8460071070 0x7f8460073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:27.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.134+0000 7f8479cd2640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8460071070 0x7f8460073530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8464005ec0 tx=0x7f8464005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:27.144 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.142+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f8470055f00 con 0x7f8474100d10 2024-09-13T15:32:27.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.315+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f84741044a0 con 0x7f8474100d10 2024-09-13T15:32:27.317 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.315+0000 7f846b7fe640 1 -- 172.21.15.90:0/3557360133 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8470059520 con 0x7f8474100d10 2024-09-13T15:32:27.318 INFO:teuthology.orchestra.run.smithi090.stdout:236223201294 2024-09-13T15:32:27.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8460071070 msgr2=0x7f8460073530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.319 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8460071070 0x7f8460073530 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8464005ec0 tx=0x7f8464005a20 comp rx=0 tx=0).stop 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 msgr2=0x7f847419b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 0x7f847419b7d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f847000b420 tx=0x7f847000b8e0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 shutdown_connections 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8460071070 0x7f8460073530 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8474101630 0x7f847419bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 --2- 172.21.15.90:0/3557360133 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8474100d10 0x7f847419b7d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 >> 172.21.15.90:0/3557360133 conn(0x7f84740758a0 msgr2=0x7f8474101f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 shutdown_connections 2024-09-13T15:32:27.320 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.317+0000 7f847c737640 1 -- 172.21.15.90:0/3557360133 wait complete. 2024-09-13T15:32:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 -- 172.21.15.90:0/2355621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 msgr2=0x7fd1a0104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2355621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a0104dd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd190009880 tx=0x7fd19002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 -- 172.21.15.90:0/2355621153 shutdown_connections 2024-09-13T15:32:27.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2355621153 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1a00fe9f0 0x7fd1a00fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2355621153 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a0104dd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 -- 172.21.15.90:0/2355621153 >> 172.21.15.90:0/2355621153 conn(0x7fd1a00758a0 msgr2=0x7fd1a00fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 -- 172.21.15.90:0/2355621153 shutdown_connections 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.321+0000 7fd1a5554640 1 -- 172.21.15.90:0/2355621153 wait complete. 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd1a5554640 1 Processor -- start 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd1a5554640 1 -- start start 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd1a5554640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1a00fe9f0 0x7fd1a019b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd1a5554640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a019bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd1a5554640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019c270 con 0x7fd1a01049f0 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd1a5554640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a01957d0 con 0x7fd1a00fe9f0 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd19e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a019bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd19e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a019bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57128/0 (socket says 172.21.15.90:57128) 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd19e7fc640 1 -- 172.21.15.90:0/2083793830 learned_addr learned my addr 172.21.15.90:0/2083793830 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:27.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.322+0000 7fd19effd640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1a00fe9f0 0x7fd1a019b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd19e7fc640 1 -- 172.21.15.90:0/2083793830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1a00fe9f0 msgr2=0x7fd1a019b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd19e7fc640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1a00fe9f0 0x7fd1a019b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd19e7fc640 1 -- 172.21.15.90:0/2083793830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd190009560 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd19e7fc640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a019bc20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd19400d6f0 tx=0x7fd19400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd194009a30 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a0195ab0 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a0196000 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd1940054b0 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.323+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd194004d00 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.324+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c005250 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.325+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd194010460 con 0x7fd1a01049f0 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.325+0000 7fd187fff640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd18806cc60 0x7fd18806f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:27.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.325+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fd19408bbc0 con 0x7fd1a01049f0 2024-09-13T15:32:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.326+0000 7fd19effd640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd18806cc60 0x7fd18806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.326+0000 7fd19effd640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd18806cc60 0x7fd18806f120 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd190006010 tx=0x7fd190005a20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:27.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.334+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd194056080 con 0x7fd1a01049f0 2024-09-13T15:32:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.504+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd18c005500 con 0x7fd1a01049f0 2024-09-13T15:32:27.506 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.504+0000 7fd187fff640 1 -- 172.21.15.90:0/2083793830 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd194020090 con 0x7fd1a01049f0 2024-09-13T15:32:27.507 INFO:teuthology.orchestra.run.smithi090.stdout:201863462930 2024-09-13T15:32:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd18806cc60 msgr2=0x7fd18806f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd18806cc60 0x7fd18806f120 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd190006010 tx=0x7fd190005a20 comp rx=0 tx=0).stop 2024-09-13T15:32:27.508 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 msgr2=0x7fd1a019bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a019bc20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd19400d6f0 tx=0x7fd19400dbb0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 shutdown_connections 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd18806cc60 0x7fd18806f120 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd1a01049f0 0x7fd1a019bc20 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 --2- 172.21.15.90:0/2083793830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1a00fe9f0 0x7fd1a019b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 >> 172.21.15.90:0/2083793830 conn(0x7fd1a00758a0 msgr2=0x7fd1a00fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 shutdown_connections 2024-09-13T15:32:27.509 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:27.506+0000 7fd1a5554640 1 -- 172.21.15.90:0/2083793830 wait complete. 2024-09-13T15:32:27.878 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201290 got 236223201294 for osd.7 2024-09-13T15:32:27.878 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:28.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:27 smithi090 ceph-mon[27811]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:28.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:27 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3557360133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T15:32:28.177 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:27 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2083793830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T15:32:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:27 smithi175 ceph-mon[36128]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:27 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3557360133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-13T15:32:28.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:27 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2083793830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-13T15:32:28.466 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462927 got 201863462930 for osd.6 2024-09-13T15:32:28.466 DEBUG:teuthology.parallel:result is None 2024-09-13T15:32:28.466 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-13T15:32:28.466 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:32:28.668 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:29 smithi090 ceph-mon[27811]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 -- 172.21.15.90:0/3842383329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 msgr2=0x7fd5d4101030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 --2- 172.21.15.90:0/3842383329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d4101030 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80098b0 tx=0x7fd5c802a4d0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 -- 172.21.15.90:0/3842383329 shutdown_connections 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 --2- 172.21.15.90:0/3842383329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5d4101600 0x7fd5d4109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 --2- 172.21.15.90:0/3842383329 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d4101030 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 -- 172.21.15.90:0/3842383329 >> 172.21.15.90:0/3842383329 conn(0x7fd5d40fa700 msgr2=0x7fd5d40fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:30.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 -- 172.21.15.90:0/3842383329 shutdown_connections 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.191+0000 7fd5db797640 1 -- 172.21.15.90:0/3842383329 wait complete. 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5db797640 1 Processor -- start 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5db797640 1 -- start start 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5db797640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d419b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5db797640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5d4101600 0x7fd5d419bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5db797640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d419c280 con 0x7fd5d4100c50 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5db797640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d41958a0 con 0x7fd5d4101600 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5d9533640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d419b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5d9533640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d419b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57146/0 (socket says 172.21.15.90:57146) 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.192+0000 7fd5d9533640 1 -- 172.21.15.90:0/2231350180 learned_addr learned my addr 172.21.15.90:0/2231350180 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5d8d32640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5d4101600 0x7fd5d419bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5d9533640 1 -- 172.21.15.90:0/2231350180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5d4101600 msgr2=0x7fd5d419bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:30.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5d9533640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5d4101600 0x7fd5d419bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5d9533640 1 -- 172.21.15.90:0/2231350180 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c8009560 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5d9533640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d419b7b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80099d0 tx=0x7fd5c8005130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c8005680 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd5c8004d40 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d4195b20 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5c803c970 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.193+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d4196010 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.194+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d4068d30 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.194+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd5c80035f0 con 0x7fd5d4100c50 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.195+0000 7fd5c67fc640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd5c006cc50 0x7fd5c006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.195+0000 7fd5d8d32640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd5c006cc50 0x7fd5c006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:30.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.195+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fd5c80ad4c0 con 0x7fd5d4100c50 2024-09-13T15:32:30.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.195+0000 7fd5d8d32640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd5c006cc50 0x7fd5c006f110 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4196d10 tx=0x7fd5d0008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:30.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.199+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd5c8077860 con 0x7fd5d4100c50 2024-09-13T15:32:30.263 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:29 smithi175 ceph-mon[36128]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:30.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.359+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5d410cc50 con 0x7fd5c006cc50 2024-09-13T15:32:30.365 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.363+0000 7fd5c67fc640 1 -- 172.21.15.90:0/2231350180 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+46306 (secure 0 0 0) 0x7fd5d410cc50 con 0x7fd5c006cc50 2024-09-13T15:32:30.365 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:32:30.366 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd5c006cc50 msgr2=0x7fd5c006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd5c006cc50 0x7fd5c006f110 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4196d10 tx=0x7fd5d0008040 comp rx=0 tx=0).stop 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 msgr2=0x7fd5d419b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d419b7b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80099d0 tx=0x7fd5c8005130 comp rx=0 tx=0).stop 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 shutdown_connections 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd5c006cc50 0x7fd5c006f110 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5d4101600 0x7fd5d419bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 --2- 172.21.15.90:0/2231350180 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd5d4100c50 0x7fd5d419b7b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 >> 172.21.15.90:0/2231350180 conn(0x7fd5d40fa700 msgr2=0x7fd5d4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 shutdown_connections 2024-09-13T15:32:30.368 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:30.365+0000 7fd5db797640 1 -- 172.21.15.90:0/2231350180 wait complete. 2024-09-13T15:32:30.993 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-13T15:32:28.550671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":222680,"kb_used_data":6496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749738536,"statfs":{"total":767960285184,"available":767732260864,"internally_reserved":0,"allocated":6651904,"data_stored":3737819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001836"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":215,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T15:31:35.357886+0000","last_change":"2024-09-13T15:31:35.357609+0000","last_active":"2024-09-13T15:31:35.357886+0000","last_peered":"2024-09-13T15:31:35.357886+0000","last_clean":"2024-09-13T15:31:35.357886+0000","last_became_active":"2024-09-13T15:31:35.357233+0000","last_became_peered":"2024-09-13T15:31:35.357233+0000","last_unstale":"2024-09-13T15:31:35.357886+0000","last_undegraded":"2024-09-13T15:31:35.357886+0000","last_fullsized":"2024-09-13T15:31:35.357886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:29:32.801761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:29:32.801761+0000","last_clean_scrub_stamp":"2024-09-13T15:29:32.801761+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:13.880937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":1,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.56999999999999995}]},{"osd":4,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.40500000000000003}]},{"osd":5,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42999999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":606208,"data_stored":245618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.69199999999999995}]},{"osd":1,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Fri Sep 13 15:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54200000000000004,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69199999999999995,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.39400000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68700000000000006,"5min":1.198,"15min":1.198},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64100000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.432}]},{"osd":3,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35999999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68999999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.378},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.35699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":245676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38,"15min":0.38200000000000001},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41699999999999998,"15min":0.41899999999999998},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Sep 13 15:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44,"15min":0.443},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.307}]},{"osd":2,"last update":"Fri Sep 13 15:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45500000000000002,"15min":0.46000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Sep 13 15:32:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.57599999999999996,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":0.502},{"interface":"front","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47499999999999998}]},{"osd":5,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.35599999999999998,"15min":0.34599999999999997},"min":{"1min":0.251,"5min":0.214,"15min":0.214},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.218}]},{"osd":1,"last update":"Fri Sep 13 15:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.36099999999999999,"15min":0.35099999999999998},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.35499999999999998,"15min":0.34300000000000003},"min":{"1min":0.316,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Sep 13 15:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.627,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.25900000000000001}]},{"osd":4,"last update":"Fri Sep 13 15:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90300000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78800000000000003,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.496}]},{"osd":5,"last update":"Fri Sep 13 15:31:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56699999999999995}]},{"osd":6,"last update":"Fri Sep 13 15:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":245676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.42099999999999999,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.436,"15min":0.46200000000000002},"min":{"1min":0.21199999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46600000000000003,"5min":4.1870000000000003,"15min":4.1870000000000003},"last":0.30499999999999999}]},{"osd":2,"last update":"Fri Sep 13 15:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.34300000000000003},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.28999999999999998}]},{"osd":3,"last update":"Fri Sep 13 15:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.44600000000000001,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64300000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.372}]},{"osd":4,"last update":"Fri Sep 13 15:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40699999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.495}]},{"osd":5,"last update":"Fri Sep 13 15:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.46200000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 15:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.316,"15min":0.29399999999999998},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45300000000000001,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.38},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.28599999999999998},"min":{"1min":0.29399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54100000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.36099999999999999}]},{"osd":2,"last update":"Fri Sep 13 15:31:38 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53600000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.34399999999999997},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.54100000000000004,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.41399999999999998}]},{"osd":3,"last update":"Fri Sep 13 15:32:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.307,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41699999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.442}]},{"osd":4,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76100000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45600000000000002}]},{"osd":5,"last update":"Fri Sep 13 15:31:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:32:30.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-13T15:32:31.192 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:31.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:31 smithi090 ceph-mon[27811]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:32:31.988 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:31 smithi090 ceph-mon[27811]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:31 smithi175 ceph-mon[36128]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:32:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:31 smithi175 ceph-mon[36128]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 -- 172.21.15.90:0/1545441129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e00ff130 msgr2=0x7fa0e00ff590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/1545441129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e00ff130 0x7fa0e00ff590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009880 tx=0x7fa0d002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 -- 172.21.15.90:0/1545441129 shutdown_connections 2024-09-13T15:32:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/1545441129 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e00ff130 0x7fa0e00ff590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/1545441129 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0104fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.718 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 -- 172.21.15.90:0/1545441129 >> 172.21.15.90:0/1545441129 conn(0x7fa0e00f9e70 msgr2=0x7fa0e00fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 -- 172.21.15.90:0/1545441129 shutdown_connections 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.703+0000 7fa0e51e5640 1 -- 172.21.15.90:0/1545441129 wait complete. 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0e51e5640 1 Processor -- start 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0e51e5640 1 -- start start 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0e51e5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e00ff130 0x7fa0e0196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0e51e5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0e51e5640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e01973c0 con 0x7fa0e0104bc0 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0e51e5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e0197530 con 0x7fa0e00ff130 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0de59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0de59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57164/0 (socket says 172.21.15.90:57164) 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.704+0000 7fa0de59c640 1 -- 172.21.15.90:0/3651345552 learned_addr learned my addr 172.21.15.90:0/3651345552 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0ded9d640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e00ff130 0x7fa0e0196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:32.719 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0de59c640 1 -- 172.21.15.90:0/3651345552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e00ff130 msgr2=0x7fa0e0196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0de59c640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e00ff130 0x7fa0e0196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0de59c640 1 -- 172.21.15.90:0/3651345552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0009560 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0de59c640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0196e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0006010 tx=0x7fa0d0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d0005460 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa0d00055c0 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e01977b0 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa0d003c900 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.705+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e019fc90 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.706+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e0100350 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.706+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa0d003cae0 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.707+0000 7fa0c7fff640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa0c806cc50 0x7fa0c806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.707+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fa0d00ad380 con 0x7fa0e0104bc0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.707+0000 7fa0ded9d640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa0c806cc50 0x7fa0c806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.707+0000 7fa0ded9d640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa0c806cc50 0x7fa0c806f110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40058c0 tx=0x7fa0d400a4c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:32.720 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.711+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa0d0077710 con 0x7fa0e0104bc0 2024-09-13T15:32:32.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.870+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0e01977b0 con 0x7fa0c806cc50 2024-09-13T15:32:32.874 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.872+0000 7fa0c7fff640 1 -- 172.21.15.90:0/3651345552 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49491 (secure 0 0 0) 0x7fa0e01977b0 con 0x7fa0c806cc50 2024-09-13T15:32:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:32:32.875 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa0c806cc50 msgr2=0x7fa0c806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa0c806cc50 0x7fa0c806f110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0d40058c0 tx=0x7fa0d400a4c0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 msgr2=0x7fa0e0196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0196e80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0006010 tx=0x7fa0d0004610 comp rx=0 tx=0).stop 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 shutdown_connections 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa0c806cc50 0x7fa0c806f110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa0e0104bc0 0x7fa0e0196e80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 --2- 172.21.15.90:0/3651345552 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e00ff130 0x7fa0e0196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 >> 172.21.15.90:0/3651345552 conn(0x7fa0e00f9e70 msgr2=0x7fa0e00fb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 shutdown_connections 2024-09-13T15:32:32.877 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:32.874+0000 7fa0e51e5640 1 -- 172.21.15.90:0/3651345552 wait complete. 2024-09-13T15:32:33.532 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-13T15:32:32.551255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":222680,"kb_used_data":6496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":749738536,"statfs":{"total":767960285184,"available":767732260864,"internally_reserved":0,"allocated":6651904,"data_stored":3737819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001826"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":215,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-13T15:31:35.357886+0000","last_change":"2024-09-13T15:31:35.357609+0000","last_active":"2024-09-13T15:31:35.357886+0000","last_peered":"2024-09-13T15:31:35.357886+0000","last_clean":"2024-09-13T15:31:35.357886+0000","last_became_active":"2024-09-13T15:31:35.357233+0000","last_became_peered":"2024-09-13T15:31:35.357233+0000","last_unstale":"2024-09-13T15:31:35.357886+0000","last_undegraded":"2024-09-13T15:31:35.357886+0000","last_fullsized":"2024-09-13T15:31:35.357886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T15:29:32.801761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T15:29:32.801761+0000","last_clean_scrub_stamp":"2024-09-13T15:29:32.801761+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:40:13.880937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28200,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716952,"statfs":{"total":95995035648,"available":95966158848,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46400000000000002}]},{"osd":2,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.441},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.505}]},{"osd":3,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58199999999999996}]},{"osd":4,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.623}]},{"osd":6,"last update":"Fri Sep 13 15:32:31 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.59899999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28204,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716948,"statfs":{"total":95995035648,"available":95966154752,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.626}]},{"osd":3,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.55300000000000005}]},{"osd":4,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.503}]},{"osd":5,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.67500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724568,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27620,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":606208,"data_stored":245618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.69199999999999995}]},{"osd":1,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Fri Sep 13 15:32:10 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27620,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717532,"statfs":{"total":95995035648,"available":95966752768,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54200000000000004,"15min":0.54900000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69199999999999995,"5min":1.2569999999999999,"15min":1.2569999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.39400000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66600000000000004,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39500000000000002}]},{"osd":1,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.36899999999999999,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68700000000000006,"5min":1.198,"15min":1.198},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.53000000000000003},"min":{"1min":0.34399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.72999999999999998,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64100000000000001,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.432}]},{"osd":3,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35999999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68999999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50800000000000001},"min":{"1min":0.39500000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.378},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"last":0.52200000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.35699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149729,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":245676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.38,"15min":0.38200000000000001},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41699999999999998,"15min":0.41899999999999998},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.28999999999999998}]},{"osd":1,"last update":"Fri Sep 13 15:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.42099999999999999,"15min":0.42899999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44,"15min":0.443},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.307}]},{"osd":2,"last update":"Fri Sep 13 15:32:00 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.45500000000000002,"15min":0.46000000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.56999999999999995,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.54700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.36699999999999999}]},{"osd":4,"last update":"Fri Sep 13 15:32:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.57599999999999996,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":0.502},{"interface":"front","average":{"1min":0.624,"5min":0.59399999999999997,"15min":0.58899999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47499999999999998}]},{"osd":5,"last update":"Fri Sep 13 15:31:44 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52300000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378664,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717536,"statfs":{"total":95995035648,"available":95966756864,"internally_reserved":0,"allocated":610304,"data_stored":245949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.35599999999999998,"15min":0.34599999999999997},"min":{"1min":0.251,"5min":0.214,"15min":0.214},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.439,"5min":0.36699999999999999,"15min":0.35499999999999998},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48499999999999999}]},{"osd":1,"last update":"Fri Sep 13 15:31:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.36099999999999999,"15min":0.35099999999999998},"min":{"1min":0.30099999999999999,"5min":0.245,"15min":0.245},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.35499999999999998,"15min":0.34300000000000003},"min":{"1min":0.316,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.501}]},{"osd":3,"last update":"Fri Sep 13 15:31:59 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.442,"15min":0.44600000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.627,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.433,"15min":0.43099999999999999},"min":{"1min":0.29399999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.443}]},{"osd":4,"last update":"Fri Sep 13 15:32:22 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90300000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78800000000000003,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.84599999999999997}]},{"osd":5,"last update":"Fri Sep 13 15:31:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Sep 13 15:32:11 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.89000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542189,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717540,"statfs":{"total":95995035648,"available":95966760960,"internally_reserved":0,"allocated":606208,"data_stored":245676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 15:32:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.42099999999999999,"15min":0.44500000000000001},"min":{"1min":0.24099999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.42099999999999999,"5min":4.0229999999999997,"15min":4.0229999999999997},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.436,"15min":0.46200000000000002},"min":{"1min":0.21199999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46600000000000003,"5min":4.1870000000000003,"15min":4.1870000000000003},"last":0.254}]},{"osd":2,"last update":"Fri Sep 13 15:31:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.35899999999999999,"15min":0.34899999999999998},"min":{"1min":0.28399999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.35399999999999998,"15min":0.34300000000000003},"min":{"1min":0.25,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.27100000000000002}]},{"osd":3,"last update":"Fri Sep 13 15:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.41099999999999998,"15min":0.41199999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.307},{"interface":"front","average":{"1min":0.435,"5min":0.44600000000000001,"15min":0.44800000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64300000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.28999999999999998}]},{"osd":4,"last update":"Fri Sep 13 15:32:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.55700000000000005,"15min":0.55300000000000005},"min":{"1min":0.40699999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.501},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.57599999999999996}]},{"osd":5,"last update":"Fri Sep 13 15:31:42 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.61499999999999999}]},{"osd":6,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.625},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":836317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 15:32:12 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.316,"15min":0.29399999999999998},"min":{"1min":0.23999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.45300000000000001,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.38},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.312,"15min":0.28599999999999998},"min":{"1min":0.29399999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54100000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"last":0.36099999999999999}]},{"osd":2,"last update":"Fri Sep 13 15:31:38 2024","interfaces":[{"interface":"back","average":{"1min":0.39600000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53600000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.35399999999999998,"15min":0.34399999999999997},"min":{"1min":0.29699999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.54100000000000004,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.41399999999999998}]},{"osd":3,"last update":"Fri Sep 13 15:32:02 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.307,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.57899999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.373},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.41699999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.442}]},{"osd":4,"last update":"Fri Sep 13 15:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.76100000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45600000000000002}]},{"osd":5,"last update":"Fri Sep 13 15:31:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Sep 13 15:32:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T15:32:33.535 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-13T15:32:33.535 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T15:32:33.535 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-13T15:32:33.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-13T15:32:33.764 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:34.128 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:33 smithi090 ceph-mon[27811]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:34.128 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:33 smithi090 ceph-mon[27811]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:32:34.128 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:34.128 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:32:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:33 smithi175 ceph-mon[36128]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:33 smithi175 ceph-mon[36128]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-13T15:32:34.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:34.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:32:35.154 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:35.154 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:32:35.154 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:35.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:35.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:32:35.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3275347964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 msgr2=0x7f9b88105fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3275347964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b88105fc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009880 tx=0x7f9b7802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3275347964 shutdown_connections 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3275347964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b880ffb30 0x7f9b880fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3275347964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b88105fc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3275347964 >> 172.21.15.90:0/3275347964 conn(0x7f9b880fb810 msgr2=0x7f9b880fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3275347964 shutdown_connections 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.335+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3275347964 wait complete. 2024-09-13T15:32:35.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b8dc4d640 1 Processor -- start 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b8dc4d640 1 -- start start 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b8dc4d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b880ffb30 0x7f9b8819b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b8dc4d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b8819bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b8dc4d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8819c0e0 con 0x7f9b88105be0 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b8dc4d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b88195700 con 0x7f9b880ffb30 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b86ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b8819bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b86ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b8819bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47474/0 (socket says 172.21.15.90:47474) 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.336+0000 7f9b86ffd640 1 -- 172.21.15.90:0/3423568942 learned_addr learned my addr 172.21.15.90:0/3423568942 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b877fe640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b880ffb30 0x7f9b8819b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b86ffd640 1 -- 172.21.15.90:0/3423568942 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b880ffb30 msgr2=0x7f9b8819b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b86ffd640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b880ffb30 0x7f9b8819b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b86ffd640 1 -- 172.21.15.90:0/3423568942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78009560 con 0x7f9b88105be0 2024-09-13T15:32:35.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b86ffd640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b8819bb50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00c8a0 tx=0x7f9b7c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7c007b80 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9b7c007ce0 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b881959e0 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7c018630 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.337+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b88195fb0 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.338+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b881011e0 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.338+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9b7c00f450 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.339+0000 7f9b84ff9640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f9b7006cc50 0x7f9b7006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.339+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f9b7c08c550 con 0x7f9b88105be0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.339+0000 7f9b877fe640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f9b7006cc50 0x7f9b7006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.339+0000 7f9b877fe640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f9b7006cc50 0x7f9b7006f110 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9b78006010 tx=0x7f9b78000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:35.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.343+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f9b7c056920 con 0x7f9b88105be0 2024-09-13T15:32:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.540+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9b880fff90 con 0x7f9b88105be0 2024-09-13T15:32:35.542 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.541+0000 7f9b84ff9640 1 -- 172.21.15.90:0/3423568942 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9b7c059f40 con 0x7f9b88105be0 2024-09-13T15:32:35.542 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:32:35.542 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T15:32:35.544 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f9b7006cc50 msgr2=0x7f9b7006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f9b7006cc50 0x7f9b7006f110 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9b78006010 tx=0x7f9b78000bc0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 msgr2=0x7f9b8819bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b8819bb50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00c8a0 tx=0x7f9b7c00cd60 comp rx=0 tx=0).stop 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 shutdown_connections 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f9b7006cc50 0x7f9b7006f110 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f9b88105be0 0x7f9b8819bb50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 --2- 172.21.15.90:0/3423568942 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b880ffb30 0x7f9b8819b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 >> 172.21.15.90:0/3423568942 conn(0x7f9b880fb810 msgr2=0x7f9b880fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 shutdown_connections 2024-09-13T15:32:35.545 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:35.543+0000 7f9b8dc4d640 1 -- 172.21.15.90:0/3423568942 wait complete. 2024-09-13T15:32:36.194 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:35 smithi090 ceph-mon[27811]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:36.194 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:35 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3423568942' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T15:32:36.216 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-13T15:32:36.216 INFO:tasks.cephadm:Setup complete, yielding 2024-09-13T15:32:36.216 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-13T15:32:36.226 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi090.front.sepia.ceph.com 2024-09-13T15:32:36.227 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph orch status' 2024-09-13T15:32:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:35 smithi175 ceph-mon[36128]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:35 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3423568942' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-13T15:32:36.427 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 -- 172.21.15.90:0/3935868456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 msgr2=0x7fb5b8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/3935868456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8101200 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8009880 tx=0x7fb5a802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 -- 172.21.15.90:0/3935868456 shutdown_connections 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/3935868456 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8101200 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/3935868456 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5b80ffb80 0x7fb5b80fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 -- 172.21.15.90:0/3935868456 >> 172.21.15.90:0/3935868456 conn(0x7fb5b80fb2f0 msgr2=0x7fb5b80fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 -- 172.21.15.90:0/3935868456 shutdown_connections 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.946+0000 7fb5c02a2640 1 -- 172.21.15.90:0/3935868456 wait complete. 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5c02a2640 1 Processor -- start 2024-09-13T15:32:37.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5c02a2640 1 -- start start 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5c02a2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5b80ffb80 0x7fb5b8195580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5c02a2640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8195ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5c02a2640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b8196000 con 0x7fb5b8100d80 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5c02a2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b8196170 con 0x7fb5b80ffb80 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5bd83d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8195ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5bd83d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8195ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47496/0 (socket says 172.21.15.90:47496) 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5bd83d640 1 -- 172.21.15.90:0/2651441046 learned_addr learned my addr 172.21.15.90:0/2651441046 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5be03e640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5b80ffb80 0x7fb5b8195580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5bd83d640 1 -- 172.21.15.90:0/2651441046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5b80ffb80 msgr2=0x7fb5b8195580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5bd83d640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5b80ffb80 0x7fb5b8195580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.947+0000 7fb5bd83d640 1 -- 172.21.15.90:0/2651441046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a8009560 con 0x7fb5b8100d80 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.948+0000 7fb5bd83d640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8195ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8005940 tx=0x7fb5a8005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:37.961 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.948+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5a8005630 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.948+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b819abf0 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.948+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb5a8003a80 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.948+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5a803cb20 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.948+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b819b0e0 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.949+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b8108640 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.949+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb5a8003bf0 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.950+0000 7fb5af7fe640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a406cc50 0x7fb5a406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.950+0000 7fb5be03e640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a406cc50 0x7fb5a406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.950+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fb5a80ad730 con 0x7fb5b8100d80 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.950+0000 7fb5be03e640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a406cc50 0x7fb5a406f110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8100be0 tx=0x7fb5b4008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:37.962 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:37.954+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb5a8077a50 con 0x7fb5b8100d80 2024-09-13T15:32:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:37 smithi090 ceph-mon[27811]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:38.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.137+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5b81056d0 con 0x7fb5a406cc50 2024-09-13T15:32:38.139 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.138+0000 7fb5af7fe640 1 -- 172.21.15.90:0/2651441046 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb5b81056d0 con 0x7fb5a406cc50 2024-09-13T15:32:38.140 INFO:teuthology.orchestra.run.smithi090.stdout:Backend: cephadm 2024-09-13T15:32:38.140 INFO:teuthology.orchestra.run.smithi090.stdout:Available: Yes 2024-09-13T15:32:38.140 INFO:teuthology.orchestra.run.smithi090.stdout:Paused: No 2024-09-13T15:32:38.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a406cc50 msgr2=0x7fb5a406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a406cc50 0x7fb5a406f110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8100be0 tx=0x7fb5b4008040 comp rx=0 tx=0).stop 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 msgr2=0x7fb5b8195ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8195ac0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8005940 tx=0x7fb5a8005230 comp rx=0 tx=0).stop 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 shutdown_connections 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a406cc50 0x7fb5a406f110 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5b8100d80 0x7fb5b8195ac0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 --2- 172.21.15.90:0/2651441046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5b80ffb80 0x7fb5b8195580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 >> 172.21.15.90:0/2651441046 conn(0x7fb5b80fb2f0 msgr2=0x7fb5b8103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 shutdown_connections 2024-09-13T15:32:38.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:38.140+0000 7fb5c02a2640 1 -- 172.21.15.90:0/2651441046 wait complete. 2024-09-13T15:32:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:37 smithi175 ceph-mon[36128]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:38.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ps' 2024-09-13T15:32:38.999 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:40.062 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:39 smithi090 ceph-mon[27811]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:40.063 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:39 smithi090 ceph-mon[27811]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:39 smithi175 ceph-mon[36128]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:39 smithi175 ceph-mon[36128]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 -- 172.21.15.90:0/2013580622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940feaa0 msgr2=0x7fb6940feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 --2- 172.21.15.90:0/2013580622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940feaa0 0x7fb6940feea0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb684009880 tx=0x7fb68402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 -- 172.21.15.90:0/2013580622 shutdown_connections 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 --2- 172.21.15.90:0/2013580622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940ffca0 0x7fb694100120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 --2- 172.21.15.90:0/2013580622 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940feaa0 0x7fb6940feea0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 -- 172.21.15.90:0/2013580622 >> 172.21.15.90:0/2013580622 conn(0x7fb6940fa250 msgr2=0x7fb6940fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 -- 172.21.15.90:0/2013580622 shutdown_connections 2024-09-13T15:32:40.521 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 -- 172.21.15.90:0/2013580622 wait complete. 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.515+0000 7fb69a142640 1 Processor -- start 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb69a142640 1 -- start start 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb69a142640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940feaa0 0x7fb694102a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb69a142640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 0x7fb694101080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb69a142640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb694102f70 con 0x7fb6940ffca0 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb69a142640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6941015f0 con 0x7fb6940feaa0 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 0x7fb694101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 0x7fb694101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47514/0 (socket says 172.21.15.90:47514) 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 -- 172.21.15.90:0/254162160 learned_addr learned my addr 172.21.15.90:0/254162160 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb6937fe640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940feaa0 0x7fb694102a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 -- 172.21.15.90:0/254162160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940feaa0 msgr2=0x7fb694102a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:40.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940feaa0 0x7fb694102a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 -- 172.21.15.90:0/254162160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb684009560 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb6937fe640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940feaa0 0x7fb694102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.516+0000 7fb692ffd640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 0x7fb694101080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb68800d6f0 tx=0x7fb68800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.517+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb688009a30 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.517+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6941018d0 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.517+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb6880054b0 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.517+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb688004d00 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.517+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb694101ea0 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.518+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6941074e0 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.518+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb688005020 con 0x7fb6940ffca0 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.519+0000 7fb690ff9640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb67c06cc50 0x7fb67c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:40.523 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.519+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fb68808be80 con 0x7fb6940ffca0 2024-09-13T15:32:40.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.519+0000 7fb6937fe640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb67c06cc50 0x7fb67c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:40.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.519+0000 7fb6937fe640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb67c06cc50 0x7fb67c06f110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb684000c00 tx=0x7fb684004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:40.524 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.523+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb6880561a0 con 0x7fb6940ffca0 2024-09-13T15:32:40.692 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.691+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb694068b60 con 0x7fb67c06cc50 2024-09-13T15:32:40.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.701+0000 7fb690ff9640 1 -- 172.21.15.90:0/254162160 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7fb694068b60 con 0x7fb67c06cc50 2024-09-13T15:32:40.702 INFO:teuthology.orchestra.run.smithi090.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:alertmanager.smithi090 smithi090 *:9093,9094 running (4m) 2m ago 5m 25.8M - 0.25.0 c8568f914cd2 6d7323d323eb 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-exporter.smithi090 smithi090 running (6m) 2m ago 6m 7176k - 18.2.4-850-g50bbe7f3 152d8bc6b19e ae2c676231e1 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-exporter.smithi175 smithi175 running (5m) 70s ago 5m 7163k - 18.2.4-850-g50bbe7f3 152d8bc6b19e 1febda9ef521 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:crash.smithi090 smithi090 running (6m) 2m ago 6m 6895k - 18.2.4-850-g50bbe7f3 152d8bc6b19e 3e10bbcbefe2 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:crash.smithi175 smithi175 running (5m) 70s ago 5m 6895k - 18.2.4-850-g50bbe7f3 152d8bc6b19e e7532659b4b8 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:grafana.smithi090 smithi090 *:3000 running (4m) 2m ago 5m 80.1M - 9.4.7 954c08fa6188 40ab4f9ea763 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.smithi090.fyvvpd smithi090 *:9283,8765,8443 running (7m) 2m ago 7m 483M - 18.2.4-850-g50bbe7f3 152d8bc6b19e f2ae25b0b003 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:mgr.smithi175.otzykk smithi175 *:8443,9283,8765 running (4m) 70s ago 4m 441M - 18.2.4-850-g50bbe7f3 152d8bc6b19e c97064ede1a6 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:mon.smithi090 smithi090 running (7m) 2m ago 7m 46.5M 2048M 18.2.4-850-g50bbe7f3 152d8bc6b19e 6e8df88e6303 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:mon.smithi175 smithi175 running (4m) 70s ago 4m 42.9M 2048M 18.2.4-850-g50bbe7f3 152d8bc6b19e b1ecf9d5db05 2024-09-13T15:32:40.703 INFO:teuthology.orchestra.run.smithi090.stdout:node-exporter.smithi090 smithi090 *:9100 running (6m) 2m ago 6m 15.3M - 1.5.0 0da6a335fe13 135784cdae56 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:node-exporter.smithi175 smithi175 *:9100 running (5m) 70s ago 5m 16.9M - 1.5.0 0da6a335fe13 43e1e1af4e1a 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 smithi090 running (4m) 2m ago 4m 47.1M 2960M 18.2.4-850-g50bbe7f3 152d8bc6b19e 53857c694096 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 smithi090 running (3m) 2m ago 3m 37.5M 2960M 18.2.4-850-g50bbe7f3 152d8bc6b19e a9705caa0b6d 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 smithi090 running (3m) 2m ago 3m 45.1M 2960M 18.2.4-850-g50bbe7f3 152d8bc6b19e 1c8532c84a7c 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 smithi090 running (2m) 2m ago 2m 12.0M 2960M 18.2.4-850-g50bbe7f3 152d8bc6b19e 0a3867cad7fc 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 smithi175 running (2m) 70s ago 2m 38.2M 3728M 18.2.4-850-g50bbe7f3 152d8bc6b19e c05efcd1485a 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 smithi175 running (2m) 70s ago 2m 45.7M 3728M 18.2.4-850-g50bbe7f3 152d8bc6b19e f733507daea4 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 smithi175 running (100s) 70s ago 99s 44.2M 3728M 18.2.4-850-g50bbe7f3 152d8bc6b19e 299ad23afa03 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 smithi175 running (75s) 70s ago 75s 11.1M 3728M 18.2.4-850-g50bbe7f3 152d8bc6b19e 773859a60aaf 2024-09-13T15:32:40.704 INFO:teuthology.orchestra.run.smithi090.stdout:prometheus.smithi090 smithi090 *:9095 running (4m) 2m ago 5m 45.0M - 2.43.0 a07b618ecd1d 68075b0df46e 2024-09-13T15:32:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb67c06cc50 msgr2=0x7fb67c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb67c06cc50 0x7fb67c06f110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb684000c00 tx=0x7fb684004f00 comp rx=0 tx=0).stop 2024-09-13T15:32:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 msgr2=0x7fb694101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 0x7fb694101080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb68800d6f0 tx=0x7fb68800dbb0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 shutdown_connections 2024-09-13T15:32:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb67c06cc50 0x7fb67c06f110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb6940ffca0 0x7fb694101080 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 --2- 172.21.15.90:0/254162160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb6940feaa0 0x7fb694102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 >> 172.21.15.90:0/254162160 conn(0x7fb6940fa250 msgr2=0x7fb694068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 shutdown_connections 2024-09-13T15:32:40.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:40.703+0000 7fb69a142640 1 -- 172.21.15.90:0/254162160 wait complete. 2024-09-13T15:32:41.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph orch ls' 2024-09-13T15:32:41.511 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:41 smithi090 ceph-mon[27811]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:41 smithi090 ceph-mon[27811]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:41 smithi175 ceph-mon[36128]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:41 smithi175 ceph-mon[36128]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 -- 172.21.15.90:0/4101052796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc100070 msgr2=0x7f23cc100470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 --2- 172.21.15.90:0/4101052796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc100070 0x7f23cc100470 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009880 tx=0x7f23bc02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 -- 172.21.15.90:0/4101052796 shutdown_connections 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 --2- 172.21.15.90:0/4101052796 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0ff150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 --2- 172.21.15.90:0/4101052796 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc100070 0x7f23cc100470 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 -- 172.21.15.90:0/4101052796 >> 172.21.15.90:0/4101052796 conn(0x7f23cc0f99b0 msgr2=0x7f23cc0fbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 -- 172.21.15.90:0/4101052796 shutdown_connections 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 -- 172.21.15.90:0/4101052796 wait complete. 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 Processor -- start 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.069+0000 7f23d2054640 1 -- start start 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23d2054640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0fe720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:43.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23d2054640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23cc100070 0x7f23cc19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23d2054640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc19a750 con 0x7f23cc0fecd0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23d2054640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc19a890 con 0x7f23cc100070 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0fe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0fe720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:47530/0 (socket says 172.21.15.90:47530) 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 -- 172.21.15.90:0/1989304398 learned_addr learned my addr 172.21.15.90:0/1989304398 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23caffd640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23cc100070 0x7f23cc19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 -- 172.21.15.90:0/1989304398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23cc100070 msgr2=0x7f23cc19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23cc100070 0x7f23cc19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 -- 172.21.15.90:0/1989304398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23bc009560 con 0x7f23cc0fecd0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23cb7fe640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0fe720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f23bc006010 tx=0x7f23bc005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc0054e0 con 0x7f23cc0fecd0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc19aae0 con 0x7f23cc0fecd0 2024-09-13T15:32:43.073 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f23bc005640 con 0x7f23cc0fecd0 2024-09-13T15:32:43.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23bc03c990 con 0x7f23cc0fecd0 2024-09-13T15:32:43.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.070+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc19b050 con 0x7f23cc0fecd0 2024-09-13T15:32:43.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.072+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc1074f0 con 0x7f23cc0fecd0 2024-09-13T15:32:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.072+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f23bc0035f0 con 0x7f23cc0fecd0 2024-09-13T15:32:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.072+0000 7f23c8ff9640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f23b406cc50 0x7f23b406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.073+0000 7f23caffd640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f23b406cc50 0x7f23b406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.073+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f23bc0ad580 con 0x7f23cc0fecd0 2024-09-13T15:32:43.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.073+0000 7f23caffd640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f23b406cc50 0x7f23b406f110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f23c0005fd0 tx=0x7f23c0005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:43.078 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.077+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f23bc0778a0 con 0x7f23cc0fecd0 2024-09-13T15:32:43.246 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.245+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f23cc104630 con 0x7f23b406cc50 2024-09-13T15:32:43.251 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.249+0000 7f23c8ff9640 1 -- 172.21.15.90:0/1989304398 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1300 (secure 0 0 0) 0x7f23cc104630 con 0x7f23b406cc50 2024-09-13T15:32:43.251 INFO:teuthology.orchestra.run.smithi090.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-13T15:32:43.251 INFO:teuthology.orchestra.run.smithi090.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-09-13T15:32:43.251 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-exporter 2/2 2m ago 6m * 2024-09-13T15:32:43.251 INFO:teuthology.orchestra.run.smithi090.stdout:crash 2/2 2m ago 6m * 2024-09-13T15:32:43.251 INFO:teuthology.orchestra.run.smithi090.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-09-13T15:32:43.252 INFO:teuthology.orchestra.run.smithi090.stdout:mgr 2/2 2m ago 6m count:2 2024-09-13T15:32:43.252 INFO:teuthology.orchestra.run.smithi090.stdout:mon 2/2 2m ago 6m smithi090:172.21.15.90=smithi090;smithi175:172.21.15.175=smithi175;count:2 2024-09-13T15:32:43.252 INFO:teuthology.orchestra.run.smithi090.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-09-13T15:32:43.252 INFO:teuthology.orchestra.run.smithi090.stdout:osd 8 2m ago - 2024-09-13T15:32:43.252 INFO:teuthology.orchestra.run.smithi090.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-09-13T15:32:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.251+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f23b406cc50 msgr2=0x7f23b406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.251+0000 7f23d2054640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f23b406cc50 0x7f23b406f110 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f23c0005fd0 tx=0x7f23c0005850 comp rx=0 tx=0).stop 2024-09-13T15:32:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.251+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 msgr2=0x7f23cc0fe720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.251+0000 7f23d2054640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0fe720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f23bc006010 tx=0x7f23bc005e50 comp rx=0 tx=0).stop 2024-09-13T15:32:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 shutdown_connections 2024-09-13T15:32:43.253 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f23b406cc50 0x7f23b406f110 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23cc100070 0x7f23cc19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 --2- 172.21.15.90:0/1989304398 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f23cc0fecd0 0x7f23cc0fe720 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 >> 172.21.15.90:0/1989304398 conn(0x7f23cc0f99b0 msgr2=0x7f23cc102f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 shutdown_connections 2024-09-13T15:32:43.254 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:43.252+0000 7f23d2054640 1 -- 172.21.15.90:0/1989304398 wait complete. 2024-09-13T15:32:43.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph orch host ls' 2024-09-13T15:32:44.055 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:44.137 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:43 smithi090 ceph-mon[27811]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:44.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:43 smithi175 ceph-mon[36128]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:45.113 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:44 smithi090 ceph-mon[27811]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:45.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:44 smithi175 ceph-mon[36128]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:45.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 -- 172.21.15.90:0/661909964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334100630 msgr2=0x7f1334102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:45.617 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 --2- 172.21.15.90:0/661909964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334100630 0x7f1334102a20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f13240098b0 tx=0x7f132402a4d0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 -- 172.21.15.90:0/661909964 shutdown_connections 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 --2- 172.21.15.90:0/661909964 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334102f60 0x7f1334105350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 --2- 172.21.15.90:0/661909964 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334100630 0x7f1334102a20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 -- 172.21.15.90:0/661909964 >> 172.21.15.90:0/661909964 conn(0x7f13340fa250 msgr2=0x7f13340fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 -- 172.21.15.90:0/661909964 shutdown_connections 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.614+0000 7f133a6f0640 1 -- 172.21.15.90:0/661909964 wait complete. 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f133a6f0640 1 Processor -- start 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f133a6f0640 1 -- start start 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f133a6f0640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334100630 0x7f1334195510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f133a6f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 0x7f1334195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f133a6f0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334195f90 con 0x7f1334102f60 2024-09-13T15:32:45.618 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f133a6f0640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334196100 con 0x7f1334100630 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.615+0000 7f13337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 0x7f1334195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13337fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 0x7f1334195a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57596/0 (socket says 172.21.15.90:57596) 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13337fe640 1 -- 172.21.15.90:0/2268321751 learned_addr learned my addr 172.21.15.90:0/2268321751 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f1333fff640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334100630 0x7f1334195510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13337fe640 1 -- 172.21.15.90:0/2268321751 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334100630 msgr2=0x7f1334195510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13337fe640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334100630 0x7f1334195510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13337fe640 1 -- 172.21.15.90:0/2268321751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1324009560 con 0x7f1334102f60 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f1333fff640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334100630 0x7f1334195510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13337fe640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 0x7f1334195a50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f132800c8f0 tx=0x7f132800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1328004410 con 0x7f1334102f60 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1328007c30 con 0x7f1334102f60 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133419ab90 con 0x7f1334102f60 2024-09-13T15:32:45.619 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1328003d80 con 0x7f1334102f60 2024-09-13T15:32:45.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.616+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133419b110 con 0x7f1334102f60 2024-09-13T15:32:45.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.617+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13341085f0 con 0x7f1334102f60 2024-09-13T15:32:45.620 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.618+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f13280077a0 con 0x7f1334102f60 2024-09-13T15:32:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.618+0000 7f13317fa640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f131c06cc50 0x7f131c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.618+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f132808c2f0 con 0x7f1334102f60 2024-09-13T15:32:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.618+0000 7f1333fff640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f131c06cc50 0x7f131c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:45.621 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.619+0000 7f1333fff640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f131c06cc50 0x7f131c06f110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1324000c00 tx=0x7f1324005e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:45.623 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.622+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1328056610 con 0x7f1334102f60 2024-09-13T15:32:45.793 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.792+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f13341039c0 con 0x7f131c06cc50 2024-09-13T15:32:45.794 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.793+0000 7f13317fa640 1 -- 172.21.15.90:0/2268321751 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f13341039c0 con 0x7f131c06cc50 2024-09-13T15:32:45.794 INFO:teuthology.orchestra.run.smithi090.stdout:HOST ADDR LABELS STATUS 2024-09-13T15:32:45.795 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090 172.21.15.90 2024-09-13T15:32:45.795 INFO:teuthology.orchestra.run.smithi090.stdout:smithi175 172.21.15.175 2024-09-13T15:32:45.795 INFO:teuthology.orchestra.run.smithi090.stdout:2 hosts in cluster 2024-09-13T15:32:45.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f131c06cc50 msgr2=0x7f131c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:45.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f131c06cc50 0x7f131c06f110 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1324000c00 tx=0x7f1324005e40 comp rx=0 tx=0).stop 2024-09-13T15:32:45.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 msgr2=0x7f1334195a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 0x7f1334195a50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f132800c8f0 tx=0x7f132800cdb0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 shutdown_connections 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f131c06cc50 0x7f131c06f110 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f1334102f60 0x7f1334195a50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 --2- 172.21.15.90:0/2268321751 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1334100630 0x7f1334195510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 >> 172.21.15.90:0/2268321751 conn(0x7f13340fa250 msgr2=0x7f13340fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 shutdown_connections 2024-09-13T15:32:45.797 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:45.795+0000 7f133a6f0640 1 -- 172.21.15.90:0/2268321751 wait complete. 2024-09-13T15:32:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:45 smithi090 ceph-mon[27811]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:45 smithi090 ceph-mon[27811]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:45 smithi175 ceph-mon[36128]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:45 smithi175 ceph-mon[36128]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:46.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph orch device ls' 2024-09-13T15:32:46.581 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:47.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:46 smithi090 ceph-mon[27811]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:47.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:46 smithi175 ceph-mon[36128]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.081+0000 7f722719d640 1 -- 172.21.15.90:0/4202222282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72200ffb90 msgr2=0x7f72200fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.081+0000 7f722719d640 1 --2- 172.21.15.90:0/4202222282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72200ffb90 0x7f72200fff90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f721c009880 tx=0x7f721c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.081+0000 7f722719d640 1 -- 172.21.15.90:0/4202222282 shutdown_connections 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.081+0000 7f722719d640 1 --2- 172.21.15.90:0/4202222282 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7220100d90 0x7f7220101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.081+0000 7f722719d640 1 --2- 172.21.15.90:0/4202222282 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f72200ffb90 0x7f72200fff90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.081+0000 7f722719d640 1 -- 172.21.15.90:0/4202222282 >> 172.21.15.90:0/4202222282 conn(0x7f72200fb360 msgr2=0x7f72200fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:48.083 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.082+0000 7f722719d640 1 -- 172.21.15.90:0/4202222282 shutdown_connections 2024-09-13T15:32:48.084 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.082+0000 7f722719d640 1 -- 172.21.15.90:0/4202222282 wait complete. 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.082+0000 7f722719d640 1 Processor -- start 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.082+0000 7f722719d640 1 -- start start 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.082+0000 7f722719d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72200ffb90 0x7f7220195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f722719d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 0x7f72201959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f722719d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7220195ef0 con 0x7f7220100d90 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f722719d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7220196060 con 0x7f72200ffb90 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 0x7f72201959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 0x7f72201959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57626/0 (socket says 172.21.15.90:57626) 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 -- 172.21.15.90:0/78427225 learned_addr learned my addr 172.21.15.90:0/78427225 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7224f39640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72200ffb90 0x7f7220195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:48.086 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 -- 172.21.15.90:0/78427225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72200ffb90 msgr2=0x7f7220195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72200ffb90 0x7f7220195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 -- 172.21.15.90:0/78427225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721c009560 con 0x7f7220100d90 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7217fff640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 0x7f72201959b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f721000c8a0 tx=0x7f721000cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7210007b80 con 0x7f7220100d90 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7210007ce0 con 0x7f7220100d90 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722019ab40 con 0x7f7220100d90 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.083+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7210018630 con 0x7f7220100d90 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.084+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722019b0c0 con 0x7f7220100d90 2024-09-13T15:32:48.087 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.085+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72201085b0 con 0x7f7220100d90 2024-09-13T15:32:48.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.085+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f721000f450 con 0x7f7220100d90 2024-09-13T15:32:48.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.085+0000 7f7215ffb640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f720806cc60 0x7f720806f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:48.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.085+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f721008c430 con 0x7f7220100d90 2024-09-13T15:32:48.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.085+0000 7f7224f39640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f720806cc60 0x7f720806f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:48.088 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.086+0000 7f7224f39640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f720806cc60 0x7f720806f120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f721c006010 tx=0x7f721c004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:48.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.090+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f72100567d0 con 0x7f7220100d90 2024-09-13T15:32:48.262 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.261+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f72201056e0 con 0x7f720806cc60 2024-09-13T15:32:48.264 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.263+0000 7f7215ffb640 1 -- 172.21.15.90:0/78427225 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f72201056e0 con 0x7f720806cc60 2024-09-13T15:32:48.264 INFO:teuthology.orchestra.run.smithi090.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-09-13T15:32:48.264 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400UK400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-09-13T15:32:48.264 INFO:teuthology.orchestra.run.smithi090.stdout:smithi175 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300M0400BGN 372G No 74s ago Has a FileSystem, LVM detected 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f720806cc60 msgr2=0x7f720806f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f720806cc60 0x7f720806f120 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f721c006010 tx=0x7f721c004f00 comp rx=0 tx=0).stop 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 msgr2=0x7f72201959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 0x7f72201959b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f721000c8a0 tx=0x7f721000cd60 comp rx=0 tx=0).stop 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 shutdown_connections 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f720806cc60 0x7f720806f120 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.266 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7220100d90 0x7f72201959b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 --2- 172.21.15.90:0/78427225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f72200ffb90 0x7f7220195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:48.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 >> 172.21.15.90:0/78427225 conn(0x7f72200fb360 msgr2=0x7f7220103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:48.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 shutdown_connections 2024-09-13T15:32:48.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:48.265+0000 7f722719d640 1 -- 172.21.15.90:0/78427225 wait complete. 2024-09-13T15:32:48.866 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:48.887 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-13T15:32:48.896 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi090.front.sepia.ceph.com 2024-09-13T15:32:48.896 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create foo' 2024-09-13T15:32:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:32:49.095 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:49.840 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:49 smithi090 ceph-mon[27811]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:49.840 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:49 smithi090 ceph-mon[27811]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:49 smithi175 ceph-mon[36128]: from='client.14502 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:49 smithi175 ceph-mon[36128]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 -- 172.21.15.90:0/1621318784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0ffca0 msgr2=0x7f105c100120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 --2- 172.21.15.90:0/1621318784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0ffca0 0x7f105c100120 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1044009880 tx=0x7f104402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 -- 172.21.15.90:0/1621318784 shutdown_connections 2024-09-13T15:32:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 --2- 172.21.15.90:0/1621318784 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0ffca0 0x7f105c100120 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 --2- 172.21.15.90:0/1621318784 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f105c0feaa0 0x7f105c0feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:50.597 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 -- 172.21.15.90:0/1621318784 >> 172.21.15.90:0/1621318784 conn(0x7f105c0fa230 msgr2=0x7f105c0fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 -- 172.21.15.90:0/1621318784 shutdown_connections 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.594+0000 7f1061e61640 1 -- 172.21.15.90:0/1621318784 wait complete. 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f1061e61640 1 Processor -- start 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f1061e61640 1 -- start start 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f1061e61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 0x7f105c194390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f1061e61640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f105c0ffca0 0x7f105c1948d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f1061e61640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c194e10 con 0x7f105c0feaa0 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f1061e61640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c194f80 con 0x7f105c0ffca0 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f105b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 0x7f105c194390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f105b7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 0x7f105c194390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:57648/0 (socket says 172.21.15.90:57648) 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f105b7fe640 1 -- 172.21.15.90:0/123257376 learned_addr learned my addr 172.21.15.90:0/123257376 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.595+0000 7f105affd640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f105c0ffca0 0x7f105c1948d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f105b7fe640 1 -- 172.21.15.90:0/123257376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f105c0ffca0 msgr2=0x7f105c1948d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:50.598 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f105b7fe640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f105c0ffca0 0x7f105c1948d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f105b7fe640 1 -- 172.21.15.90:0/123257376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044009560 con 0x7f105c0feaa0 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f105b7fe640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 0x7f105c194390 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f104c00ea20 tx=0x7f104c00eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f104c00cdb0 con 0x7f105c0feaa0 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f104c003e00 con 0x7f105c0feaa0 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f104c011470 con 0x7f105c0feaa0 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c199a60 con 0x7f105c0feaa0 2024-09-13T15:32:50.599 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.596+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c199f50 con 0x7f105c0feaa0 2024-09-13T15:32:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.597+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105c1074e0 con 0x7f105c0feaa0 2024-09-13T15:32:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.597+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f104c003970 con 0x7f105c0feaa0 2024-09-13T15:32:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.598+0000 7f1058ff9640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f104806cc50 0x7f104806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.598+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5502+0+0 (secure 0 0 0) 0x7f104c08c6d0 con 0x7f105c0feaa0 2024-09-13T15:32:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.598+0000 7f105affd640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f104806cc50 0x7f104806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.598+0000 7f105affd640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f104806cc50 0x7f104806f110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1044000c00 tx=0x7f1044004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:50.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.602+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f104c056b20 con 0x7f105c0feaa0 2024-09-13T15:32:50.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:50.786+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}) v1 -- 0x7f105c19a220 con 0x7f104806cc50 2024-09-13T15:32:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:51 smithi090 ceph-mon[27811]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:51 smithi090 ceph-mon[27811]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:51 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T15:32:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:51 smithi175 ceph-mon[36128]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:51 smithi175 ceph-mon[36128]: from='client.14506 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:51 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-13T15:32:52.656 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.655+0000 7f1058ff9640 1 -- 172.21.15.90:0/123257376 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f105c19a220 con 0x7f104806cc50 2024-09-13T15:32:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f104806cc50 msgr2=0x7f104806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f104806cc50 0x7f104806f110 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1044000c00 tx=0x7f1044004f00 comp rx=0 tx=0).stop 2024-09-13T15:32:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 msgr2=0x7f105c194390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 0x7f105c194390 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f104c00ea20 tx=0x7f104c00eee0 comp rx=0 tx=0).stop 2024-09-13T15:32:52.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 shutdown_connections 2024-09-13T15:32:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f104806cc50 0x7f104806f110 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f105c0ffca0 0x7f105c1948d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 --2- 172.21.15.90:0/123257376 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f105c0feaa0 0x7f105c194390 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 >> 172.21.15.90:0/123257376 conn(0x7f105c0fa230 msgr2=0x7f105c102ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 shutdown_connections 2024-09-13T15:32:52.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:52.658+0000 7f1061e61640 1 -- 172.21.15.90:0/123257376 wait complete. 2024-09-13T15:32:53.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:52 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T15:32:53.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:52 smithi090 ceph-mon[27811]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T15:32:53.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:52 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T15:32:53.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:52 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090[27807]: 2024-09-13T15:32:52.636+0000 7f24be263640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T15:32:53.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:52 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-13T15:32:53.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:52 smithi175 ceph-mon[36128]: osdmap e60: 8 total, 8 up, 8 in 2024-09-13T15:32:53.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:52 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-13T15:32:53.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- bash -c 'ceph fs volume create bar' 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: pgmap v215: 33 pgs: 5 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: fsmap foo:0 2024-09-13T15:32:54.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: Saving service mds.foo spec with placement count:2 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.zyijng", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.zyijng", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:54.003 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:53 smithi090 ceph-mon[27811]: Deploying daemon mds.foo.smithi090.zyijng on smithi090 2024-09-13T15:32:54.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: pgmap v215: 33 pgs: 5 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: osdmap e61: 8 total, 8 up, 8 in 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: osdmap e62: 8 total, 8 up, 8 in 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: fsmap foo:0 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: Saving service mds.foo spec with placement count:2 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:54.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.zyijng", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi090.zyijng", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:53 smithi175 ceph-mon[36128]: Deploying daemon mds.foo.smithi090.zyijng on smithi090 2024-09-13T15:32:54.644 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:32:54.857 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:54 smithi090 ceph-mon[27811]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T15:32:55.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:54 smithi175 ceph-mon[36128]: osdmap e63: 8 total, 8 up, 8 in 2024-09-13T15:32:55.995 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:55 smithi090 ceph-mon[27811]: pgmap v219: 65 pgs: 7 creating+activating, 22 creating+peering, 18 unknown, 18 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:55.995 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:55 smithi090 ceph-mon[27811]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:32:55.995 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:55 smithi090 ceph-mon[27811]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T15:32:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:55 smithi175 ceph-mon[36128]: pgmap v219: 65 pgs: 7 creating+activating, 22 creating+peering, 18 unknown, 18 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:55 smithi175 ceph-mon[36128]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:32:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:55 smithi175 ceph-mon[36128]: osdmap e64: 8 total, 8 up, 8 in 2024-09-13T15:32:56.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.553+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1779648135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80ffcf0 msgr2=0x7fe1c8100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:56.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.553+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1779648135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c8100170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009c20 tx=0x7fe1bc02a970 comp rx=0 tx=0).stop 2024-09-13T15:32:56.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1779648135 shutdown_connections 2024-09-13T15:32:56.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1779648135 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c8100170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:56.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1779648135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80feef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:56.815 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1779648135 >> 172.21.15.90:0/1779648135 conn(0x7fe1c80fa280 msgr2=0x7fe1c80fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1779648135 shutdown_connections 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1779648135 wait complete. 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.554+0000 7fe1ccb53640 1 Processor -- start 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1ccb53640 1 -- start start 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1ccb53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80687d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1ccb53640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1ccb53640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c819a790 con 0x7fe1c80feaf0 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1ccb53640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c819a900 con 0x7fe1c80ffcf0 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80687d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37816/0 (socket says 172.21.15.90:37816) 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 -- 172.21.15.90:0/1851110467 learned_addr learned my addr 172.21.15.90:0/1851110467 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:56.816 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 -- 172.21.15.90:0/1851110467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80ffcf0 msgr2=0x7fe1c819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c5d9b640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 -- 172.21.15.90:0/1851110467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc009900 con 0x7fe1c80feaf0 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c5d9b640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1c659c640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80687d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000d8d0 tx=0x7fe1c000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.555+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1c0009a30 con 0x7fe1c80feaf0 2024-09-13T15:32:56.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.556+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c819abe0 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.556+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe1c0009b90 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.556+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1c000f5f0 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.556+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c819b130 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.556+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c8107530 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.557+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe1c0009d00 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.557+0000 7fe1b77fe640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe1b006cc50 0x7fe1b006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.558+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6320+0+0 (secure 0 0 0) 0x7fe1c008c4b0 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.558+0000 7fe1c5d9b640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe1b006cc50 0x7fe1b006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.558+0000 7fe1c5d9b640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe1b006cc50 0x7fe1b006f110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc006010 tx=0x7fe1bc009890 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.562+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fe1c00564a0 con 0x7fe1c80feaf0 2024-09-13T15:32:56.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:56.768+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 --> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1c819b320 con 0x7fe1b006cc50 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: pgmap v221: 65 pgs: 7 creating+activating, 22 creating+peering, 6 unknown, 30 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] up:boot 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: daemon mds.foo.smithi090.zyijng assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: Cluster is now healthy 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: fsmap foo:0 1 up:standby 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "foo.smithi090.zyijng"}]: dispatch 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: fsmap foo:1 {0=foo.smithi090.zyijng=up:creating} 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: daemon mds.foo.smithi090.zyijng is now active in filesystem foo as rank 0 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:57.986 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi175.cvvnzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi175.cvvnzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: Deploying daemon mds.foo.smithi175.cvvnzp on smithi175 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:57.987 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:57 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]: dispatch 2024-09-13T15:32:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: pgmap v221: 65 pgs: 7 creating+activating, 22 creating+peering, 6 unknown, 30 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] up:boot 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: daemon mds.foo.smithi090.zyijng assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: Cluster is now healthy 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: fsmap foo:0 1 up:standby 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "foo.smithi090.zyijng"}]: dispatch 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: fsmap foo:1 {0=foo.smithi090.zyijng=up:creating} 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: daemon mds.foo.smithi090.zyijng is now active in filesystem foo as rank 0 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:58.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi175.cvvnzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi175.cvvnzp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: Deploying daemon mds.foo.smithi175.cvvnzp on smithi175 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='client.14510 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "bar", "target": ["mon-mgr", ""]}]: dispatch 2024-09-13T15:32:58.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:57 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]: dispatch 2024-09-13T15:32:58.765 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.762+0000 7fe1b77fe640 1 -- 172.21.15.90:0/1851110467 <== mgr.14217 v2:172.21.15.90:6800/885641711 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe1c819b320 con 0x7fe1b006cc50 2024-09-13T15:32:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe1b006cc50 msgr2=0x7fe1b006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe1b006cc50 0x7fe1b006f110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc006010 tx=0x7fe1bc009890 comp rx=0 tx=0).stop 2024-09-13T15:32:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 msgr2=0x7fe1c80687d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80687d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000d8d0 tx=0x7fe1c000dd90 comp rx=0 tx=0).stop 2024-09-13T15:32:58.766 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 shutdown_connections 2024-09-13T15:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe1b006cc50 0x7fe1b006f110 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe1c80ffcf0 0x7fe1c819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 --2- 172.21.15.90:0/1851110467 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe1c80feaf0 0x7fe1c80687d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 >> 172.21.15.90:0/1851110467 conn(0x7fe1c80fa280 msgr2=0x7fe1c8102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 shutdown_connections 2024-09-13T15:32:58.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:58.765+0000 7fe1ccb53640 1 -- 172.21.15.90:0/1851110467 wait complete. 2024-09-13T15:32:58.953 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:58 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] up:active 2024-09-13T15:32:58.953 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:58 smithi090 ceph-mon[27811]: fsmap foo:1 {0=foo.smithi090.zyijng=up:active} 2024-09-13T15:32:58.953 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:58 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]': finished 2024-09-13T15:32:58.953 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:58 smithi090 ceph-mon[27811]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T15:32:58.953 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:58 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]: dispatch 2024-09-13T15:32:58.953 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:58 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090[27807]: 2024-09-13T15:32:58.742+0000 7f24be263640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T15:32:58.957 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:58 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] up:active 2024-09-13T15:32:58.957 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:58 smithi175 ceph-mon[36128]: fsmap foo:1 {0=foo.smithi090.zyijng=up:active} 2024-09-13T15:32:58.957 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:58 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pool create", "pool": "cephfs.bar.meta"}]': finished 2024-09-13T15:32:58.957 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:58 smithi175 ceph-mon[36128]: osdmap e65: 8 total, 8 up, 8 in 2024-09-13T15:32:58.957 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:58 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]: dispatch 2024-09-13T15:32:59.483 INFO:teuthology.run_tasks:Running task fs.ready... 2024-09-13T15:32:59.496 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:32:59.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 -- 172.21.15.90:0/4215354983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 msgr2=0x7f12800fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:59.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 --2- 172.21.15.90:0/4215354983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f12800fed90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f1274009880 tx=0x7f127402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 -- 172.21.15.90:0/4215354983 shutdown_connections 2024-09-13T15:32:59.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 --2- 172.21.15.90:0/4215354983 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12800ffb80 0x7f1280100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 --2- 172.21.15.90:0/4215354983 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f12800fed90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 -- 172.21.15.90:0/4215354983 >> 172.21.15.90:0/4215354983 conn(0x7f12800fa0f0 msgr2=0x7f12800fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 -- 172.21.15.90:0/4215354983 shutdown_connections 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.603+0000 7f1288118640 1 -- 172.21.15.90:0/4215354983 wait complete. 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1288118640 1 Processor -- start 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1288118640 1 -- start start 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1288118640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f1280195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1288118640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12800ffb80 0x7f1280195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1288118640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280195ea0 con 0x7f12800fe990 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1288118640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1280196010 con 0x7f12800ffb80 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f1280195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f1280195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37838/0 (socket says 172.21.15.90:37838) 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 -- 172.21.15.90:0/13891349 learned_addr learned my addr 172.21.15.90:0/13891349 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:59.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f12856b3640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12800ffb80 0x7f1280195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 -- 172.21.15.90:0/13891349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12800ffb80 msgr2=0x7f1280195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12800ffb80 0x7f1280195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 -- 172.21.15.90:0/13891349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1274009560 con 0x7f12800fe990 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.604+0000 7f1285eb4640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f1280195420 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1274006010 tx=0x7f1274005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.605+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12740054e0 con 0x7f12800fe990 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.605+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1274005640 con 0x7f12800fe990 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.605+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128019aa80 con 0x7f12800fe990 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.605+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f127403c990 con 0x7f12800fe990 2024-09-13T15:32:59.608 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.605+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128019afd0 con 0x7f12800fe990 2024-09-13T15:32:59.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.606+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12801073c0 con 0x7f12800fe990 2024-09-13T15:32:59.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.606+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f12740035f0 con 0x7f12800fe990 2024-09-13T15:32:59.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.607+0000 7f1272ffd640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f126c06cc50 0x7f126c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:59.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.607+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f12740adb20 con 0x7f12800fe990 2024-09-13T15:32:59.609 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.607+0000 7f12856b3640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f126c06cc50 0x7f126c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:59.610 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.607+0000 7f12856b3640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f126c06cc50 0x7f126c06f110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1280196930 tx=0x7f127c00b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:59.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.611+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1274077880 con 0x7f12800fe990 2024-09-13T15:32:59.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.808+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f128019b450 con 0x7f12800fe990 2024-09-13T15:32:59.812 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.810+0000 7f1272ffd640 1 -- 172.21.15.90:0/13891349 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+21070 (secure 0 0 0) 0x7f127407aea0 con 0x7f12800fe990 2024-09-13T15:32:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:32:59.812 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":68,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:32:59.745985+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":163841,"flags_names":"hashpspool,creating,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f126c06cc50 msgr2=0x7f126c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f126c06cc50 0x7f126c06f110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1280196930 tx=0x7f127c00b390 comp rx=0 tx=0).stop 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 msgr2=0x7f1280195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f1280195420 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1274006010 tx=0x7f1274005e50 comp rx=0 tx=0).stop 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 shutdown_connections 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f126c06cc50 0x7f126c06f110 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12800ffb80 0x7f1280195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 --2- 172.21.15.90:0/13891349 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12800fe990 0x7f1280195420 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 >> 172.21.15.90:0/13891349 conn(0x7f12800fa0f0 msgr2=0x7f1280067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 shutdown_connections 2024-09-13T15:32:59.814 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.812+0000 7f1288118640 1 -- 172.21.15.90:0/13891349 wait complete. 2024-09-13T15:32:59.826 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 163841, 'flags_names': 'hashpspool,creating,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:32:59.826 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 -- 172.21.15.90:0/2474552992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 msgr2=0x7fcc680fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 --2- 172.21.15.90:0/2474552992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc680fed70 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009880 tx=0x7fcc6402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 -- 172.21.15.90:0/2474552992 shutdown_connections 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 --2- 172.21.15.90:0/2474552992 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc680ffb60 0x7fcc680fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 --2- 172.21.15.90:0/2474552992 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc680fed70 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 -- 172.21.15.90:0/2474552992 >> 172.21.15.90:0/2474552992 conn(0x7fcc680fa0f0 msgr2=0x7fcc680fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:32:59.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 -- 172.21.15.90:0/2474552992 shutdown_connections 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 -- 172.21.15.90:0/2474552992 wait complete. 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 Processor -- start 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.931+0000 7fcc703ed640 1 -- start start 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc703ed640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc68194300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc703ed640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc680ffb60 0x7fcc68194840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc703ed640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68194d80 con 0x7fcc680fe970 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc703ed640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc68194ef0 con 0x7fcc680ffb60 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc68194300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc68194300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37862/0 (socket says 172.21.15.90:37862) 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 -- 172.21.15.90:0/542862951 learned_addr learned my addr 172.21.15.90:0/542862951 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6d988640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc680ffb60 0x7fcc68194840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 -- 172.21.15.90:0/542862951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc680ffb60 msgr2=0x7fcc68194840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:32:59.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc680ffb60 0x7fcc68194840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 -- 172.21.15.90:0/542862951 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc64009560 con 0x7fcc680fe970 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc6e189640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc68194300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc64006010 tx=0x7fcc64005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc640054e0 con 0x7fcc680fe970 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcc64005640 con 0x7fcc680fe970 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc68199960 con 0x7fcc680fe970 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcc6403c990 con 0x7fcc680fe970 2024-09-13T15:32:59.936 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.932+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc68199eb0 con 0x7fcc680fe970 2024-09-13T15:32:59.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.933+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc681073a0 con 0x7fcc680fe970 2024-09-13T15:32:59.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.934+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcc640035f0 con 0x7fcc680fe970 2024-09-13T15:32:59.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.935+0000 7fcc5f7fe640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcc5406cc50 0x7fcc5406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:32:59.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.935+0000 7fcc6d988640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcc5406cc50 0x7fcc5406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:32:59.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.935+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fcc640ada50 con 0x7fcc680fe970 2024-09-13T15:32:59.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.935+0000 7fcc6d988640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcc5406cc50 0x7fcc5406f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc68195810 tx=0x7fcc5800b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:32:59.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:32:59.939+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fcc64077830 con 0x7fcc680fe970 2024-09-13T15:33:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: pgmap v223: 97 pgs: 11 unknown, 21 creating+peering, 1 creating+activating, 64 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-09-13T15:33:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:33:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]': finished 2024-09-13T15:33:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]: dispatch 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]': finished 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: fsmap foo:1 bar:0 {foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: Saving service mds.bar spec with placement count:2 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:32:59 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:33:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: pgmap v223: 97 pgs: 11 unknown, 21 creating+peering, 1 creating+activating, 64 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-09-13T15:33:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.bar.data"}]': finished 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: osdmap e66: 8 total, 8 up, 8 in 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]: dispatch 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: osdmap e67: 8 total, 8 up, 8 in 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "fs new", "fs_name": "bar", "metadata": "cephfs.bar.meta", "data": "cephfs.bar.data"}]': finished 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: fsmap foo:1 bar:0 {foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:00.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: Saving service mds.bar spec with placement count:2 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:00.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:32:59 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:33:00.111 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.110+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc6819a430 con 0x7fcc680fe970 2024-09-13T15:33:00.112 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.110+0000 7fcc5f7fe640 1 -- 172.21.15.90:0/542862951 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc64040020 con 0x7fcc680fe970 2024-09-13T15:33:00.112 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:33:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcc5406cc50 msgr2=0x7fcc5406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcc5406cc50 0x7fcc5406f110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcc68195810 tx=0x7fcc5800b390 comp rx=0 tx=0).stop 2024-09-13T15:33:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 msgr2=0x7fcc68194300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc68194300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc64006010 tx=0x7fcc64005e50 comp rx=0 tx=0).stop 2024-09-13T15:33:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 shutdown_connections 2024-09-13T15:33:00.114 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcc5406cc50 0x7fcc5406f110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcc680ffb60 0x7fcc68194840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 --2- 172.21.15.90:0/542862951 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcc680fe970 0x7fcc68194300 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 >> 172.21.15.90:0/542862951 conn(0x7fcc680fa0f0 msgr2=0x7fcc68102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 shutdown_connections 2024-09-13T15:33:00.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.112+0000 7fcc703ed640 1 -- 172.21.15.90:0/542862951 wait complete. 2024-09-13T15:33:00.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:33:00.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 -- 172.21.15.90:0/2515909914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4684100c80 msgr2=0x7f4684101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 --2- 172.21.15.90:0/2515909914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4684100c80 0x7f4684101100 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f4678009880 tx=0x7f467802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 -- 172.21.15.90:0/2515909914 shutdown_connections 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 --2- 172.21.15.90:0/2515909914 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f4684100c80 0x7f4684101100 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 --2- 172.21.15.90:0/2515909914 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f46840ffa90 0x7f46840ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 -- 172.21.15.90:0/2515909914 >> 172.21.15.90:0/2515909914 conn(0x7f46840fb1f0 msgr2=0x7f46840fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 -- 172.21.15.90:0/2515909914 shutdown_connections 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 -- 172.21.15.90:0/2515909914 wait complete. 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.226+0000 7f468c60d640 1 Processor -- start 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 -- start start 2024-09-13T15:33:00.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 0x7f4684195590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4684100c80 0x7f4684195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684196010 con 0x7f46840ffa90 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684196180 con 0x7f4684100c80 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 0x7f4684195590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 0x7f4684195590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37876/0 (socket says 172.21.15.90:37876) 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 -- 172.21.15.90:0/1343455287 learned_addr learned my addr 172.21.15.90:0/1343455287 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f4689ba8640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4684100c80 0x7f4684195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 -- 172.21.15.90:0/1343455287 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4684100c80 msgr2=0x7f4684195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4684100c80 0x7f4684195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 -- 172.21.15.90:0/1343455287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678009560 con 0x7f46840ffa90 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f4689ba8640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4684100c80 0x7f4684195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468a3a9640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 0x7f4684195590 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f468000b3d0 tx=0x7f468000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:00.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4680011800 con 0x7f46840ffa90 2024-09-13T15:33:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4680011df0 con 0x7f46840ffa90 2024-09-13T15:33:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468419ac50 con 0x7f46840ffa90 2024-09-13T15:33:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f468000f2e0 con 0x7f46840ffa90 2024-09-13T15:33:00.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.227+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468419b180 con 0x7f46840ffa90 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.228+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46840fff10 con 0x7f46840ffa90 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.229+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4680011960 con 0x7f46840ffa90 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.229+0000 7f46777fe640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f467006cc50 0x7f467006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.229+0000 7f4689ba8640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f467006cc50 0x7f467006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.229+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f468008c590 con 0x7f46840ffa90 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.230+0000 7f4689ba8640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f467006cc50 0x7f467006f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4678000c00 tx=0x7f4678005e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:00.236 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.233+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f46800562c0 con 0x7f46840ffa90 2024-09-13T15:33:00.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.401+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7f468419b450 con 0x7f46840ffa90 2024-09-13T15:33:00.402 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.401+0000 7f46777fe640 1 -- 172.21.15.90:0/1343455287 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v68) v1 ==== 104+0+11 (secure 0 0 0) 0x7f46800598e0 con 0x7f46840ffa90 2024-09-13T15:33:00.402 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:00.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f467006cc50 msgr2=0x7f467006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.404 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f467006cc50 0x7f467006f110 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4678000c00 tx=0x7f4678005e40 comp rx=0 tx=0).stop 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 msgr2=0x7f4684195590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 0x7f4684195590 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f468000b3d0 tx=0x7f468000b890 comp rx=0 tx=0).stop 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 shutdown_connections 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f467006cc50 0x7f467006f110 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4684100c80 0x7f4684195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 --2- 172.21.15.90:0/1343455287 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f46840ffa90 0x7f4684195590 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 >> 172.21.15.90:0/1343455287 conn(0x7f46840fb1f0 msgr2=0x7f4684103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 shutdown_connections 2024-09-13T15:33:00.405 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.403+0000 7f468c60d640 1 -- 172.21.15.90:0/1343455287 wait complete. 2024-09-13T15:33:00.415 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:33:00.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 -- 172.21.15.90:0/2977979319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 msgr2=0x7fd018101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2977979319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018101120 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd008009880 tx=0x7fd00802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 -- 172.21.15.90:0/2977979319 shutdown_connections 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2977979319 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018101120 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2977979319 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180ffab0 0x7fd0180ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 -- 172.21.15.90:0/2977979319 >> 172.21.15.90:0/2977979319 conn(0x7fd0180fb250 msgr2=0x7fd0180fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 -- 172.21.15.90:0/2977979319 shutdown_connections 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.522+0000 7fd01f30f640 1 -- 172.21.15.90:0/2977979319 wait complete. 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01f30f640 1 Processor -- start 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01f30f640 1 -- start start 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01f30f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180ffab0 0x7fd018195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01f30f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01f30f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd018195ec0 con 0x7fd018100ca0 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01f30f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd018196030 con 0x7fd0180ffab0 2024-09-13T15:33:00.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01c8aa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01c8aa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37900/0 (socket says 172.21.15.90:37900) 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01c8aa640 1 -- 172.21.15.90:0/2817348217 learned_addr learned my addr 172.21.15.90:0/2817348217 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.523+0000 7fd01d0ab640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180ffab0 0x7fd018195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd01c8aa640 1 -- 172.21.15.90:0/2817348217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180ffab0 msgr2=0x7fd018195440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd01c8aa640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180ffab0 0x7fd018195440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd01c8aa640 1 -- 172.21.15.90:0/2817348217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008009560 con 0x7fd018100ca0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd01c8aa640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018195980 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd008004f60 tx=0x7fd008004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0080054e0 con 0x7fd018100ca0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd008005640 con 0x7fd018100ca0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd01819aaa0 con 0x7fd018100ca0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd00803c960 con 0x7fd018100ca0 2024-09-13T15:33:00.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.524+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd01819aff0 con 0x7fd018100ca0 2024-09-13T15:33:00.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.525+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0180ffeb0 con 0x7fd018100ca0 2024-09-13T15:33:00.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.526+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd008003cf0 con 0x7fd018100ca0 2024-09-13T15:33:00.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.526+0000 7fd00e7fc640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd00406cc50 0x7fd00406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.526+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fd0080adac0 con 0x7fd018100ca0 2024-09-13T15:33:00.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.526+0000 7fd01d0ab640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd00406cc50 0x7fd00406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.527+0000 7fd01d0ab640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd00406cc50 0x7fd00406f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd018100b00 tx=0x7fd01400b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:00.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.531+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd008077820 con 0x7fd018100ca0 2024-09-13T15:33:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.710+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7fd0181086a0 con 0x7fd018100ca0 2024-09-13T15:33:00.711 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.710+0000 7fd00e7fc640 1 -- 172.21.15.90:0/2817348217 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v68) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd00807ae40 con 0x7fd018100ca0 2024-09-13T15:33:00.711 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd00406cc50 msgr2=0x7fd00406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd00406cc50 0x7fd00406f110 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd018100b00 tx=0x7fd01400b390 comp rx=0 tx=0).stop 2024-09-13T15:33:00.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 msgr2=0x7fd018195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018195980 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd008004f60 tx=0x7fd008004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 shutdown_connections 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd00406cc50 0x7fd00406f110 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd018100ca0 0x7fd018195980 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 --2- 172.21.15.90:0/2817348217 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0180ffab0 0x7fd018195440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 >> 172.21.15.90:0/2817348217 conn(0x7fd0180fb250 msgr2=0x7fd018103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 shutdown_connections 2024-09-13T15:33:00.714 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.712+0000 7fd01f30f640 1 -- 172.21.15.90:0/2817348217 wait complete. 2024-09-13T15:33:00.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:33:00.828 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.825+0000 7f6a57072640 1 -- 172.21.15.90:0/4214421715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 msgr2=0x7f6a500fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.825+0000 7f6a57072640 1 --2- 172.21.15.90:0/4214421715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a500fed90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009880 tx=0x7f6a4802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- 172.21.15.90:0/4214421715 shutdown_connections 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 --2- 172.21.15.90:0/4214421715 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a500ffb80 0x7f6a50100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 --2- 172.21.15.90:0/4214421715 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a500fed90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- 172.21.15.90:0/4214421715 >> 172.21.15.90:0/4214421715 conn(0x7f6a500fa0f0 msgr2=0x7f6a500fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- 172.21.15.90:0/4214421715 shutdown_connections 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- 172.21.15.90:0/4214421715 wait complete. 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 Processor -- start 2024-09-13T15:33:00.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- start start 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a50102920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a500ffb80 0x7f6a50100f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50102e60 con 0x7f6a500fe990 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a57072640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a501014d0 con 0x7f6a500ffb80 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a54e0e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a50102920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a54e0e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a50102920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37922/0 (socket says 172.21.15.90:37922) 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.826+0000 7f6a54e0e640 1 -- 172.21.15.90:0/1061371609 learned_addr learned my addr 172.21.15.90:0/1061371609 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a4ffff640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a500ffb80 0x7f6a50100f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a54e0e640 1 -- 172.21.15.90:0/1061371609 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a500ffb80 msgr2=0x7f6a50100f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a54e0e640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a500ffb80 0x7f6a50100f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a54e0e640 1 -- 172.21.15.90:0/1061371609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48009560 con 0x7f6a500fe990 2024-09-13T15:33:00.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a54e0e640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a50102920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009230 tx=0x7f6a48004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:00.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a480054e0 con 0x7f6a500fe990 2024-09-13T15:33:00.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6a48005640 con 0x7f6a500fe990 2024-09-13T15:33:00.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a50101750 con 0x7f6a500fe990 2024-09-13T15:33:00.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a4803c970 con 0x7f6a500fe990 2024-09-13T15:33:00.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.827+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a50101cd0 con 0x7f6a500fe990 2024-09-13T15:33:00.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.828+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a500fed90 con 0x7f6a500fe990 2024-09-13T15:33:00.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.829+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6a480035f0 con 0x7f6a500fe990 2024-09-13T15:33:00.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.829+0000 7f6a4dffb640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6a3c06cc50 0x7f6a3c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:00.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.829+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f6a480adb90 con 0x7f6a500fe990 2024-09-13T15:33:00.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.829+0000 7f6a4ffff640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6a3c06cc50 0x7f6a3c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:00.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.830+0000 7f6a4ffff640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6a3c06cc50 0x7f6a3c06f110 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a50102690 tx=0x7f6a38009680 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:00.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:00.833+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6a480778f0 con 0x7f6a500fe990 2024-09-13T15:33:00.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T15:33:00.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] up:boot 2024-09-13T15:33:00.837 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: daemon mds.foo.smithi175.cvvnzp assigned to filesystem bar as rank 0 (now has 1 ranks) 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: fsmap foo:1 bar:0 {foo:0=foo.smithi090.zyijng=up:active} 1 up:standby 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "foo.smithi175.cvvnzp"}]: dispatch 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1 {bar:0=foo.smithi175.cvvnzp=up:creating,foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: daemon mds.foo.smithi175.cvvnzp is now active in filesystem bar as rank 0 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/13891349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/542862951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:00.838 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1343455287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:00.839 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:00 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2817348217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:00.982 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: osdmap e68: 8 total, 8 up, 8 in 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] up:boot 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: daemon mds.foo.smithi175.cvvnzp assigned to filesystem bar as rank 0 (now has 1 ranks) 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: fsmap foo:1 bar:0 {foo:0=foo.smithi090.zyijng=up:active} 1 up:standby 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "foo.smithi175.cvvnzp"}]: dispatch 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1 {bar:0=foo.smithi175.cvvnzp=up:creating,foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: daemon mds.foo.smithi175.cvvnzp is now active in filesystem bar as rank 0 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/13891349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:00.983 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/542862951' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:00.984 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1343455287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:00.984 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:00 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2817348217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:01.009 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.008+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7f6a501073c0 con 0x7f6a500fe990 2024-09-13T15:33:01.010 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.009+0000 7f6a4dffb640 1 -- 172.21.15.90:0/1061371609 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6a4807af10 con 0x7f6a500fe990 2024-09-13T15:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:01.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6a3c06cc50 msgr2=0x7f6a3c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.012 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6a3c06cc50 0x7f6a3c06f110 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6a50102690 tx=0x7f6a38009680 comp rx=0 tx=0).stop 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 msgr2=0x7f6a50102920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a50102920 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6a48009230 tx=0x7f6a48004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 shutdown_connections 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6a3c06cc50 0x7f6a3c06f110 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a500ffb80 0x7f6a50100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 --2- 172.21.15.90:0/1061371609 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6a500fe990 0x7f6a50102920 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 >> 172.21.15.90:0/1061371609 conn(0x7f6a500fa0f0 msgr2=0x7f6a50067ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 shutdown_connections 2024-09-13T15:33:01.013 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.011+0000 7f6a57072640 1 -- 172.21.15.90:0/1061371609 wait complete. 2024-09-13T15:33:01.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:33:01.127 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 -- 172.21.15.90:0/3049670345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 msgr2=0x7fc124101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 --2- 172.21.15.90:0/3049670345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc124101100 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc118009880 tx=0x7fc11802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 -- 172.21.15.90:0/3049670345 shutdown_connections 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 --2- 172.21.15.90:0/3049670345 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc124101100 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 --2- 172.21.15.90:0/3049670345 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 0x7fc1240ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 -- 172.21.15.90:0/3049670345 >> 172.21.15.90:0/3049670345 conn(0x7fc1240fb1f0 msgr2=0x7fc1240fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 -- 172.21.15.90:0/3049670345 shutdown_connections 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.124+0000 7fc129e61640 1 -- 172.21.15.90:0/3049670345 wait complete. 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc129e61640 1 Processor -- start 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc129e61640 1 -- start start 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc129e61640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 0x7fc124195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc129e61640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc1241959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.128 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc129e61640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124195ef0 con 0x7fc124100c80 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc129e61640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc124196060 con 0x7fc1240ffa90 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc122ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc1241959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc122ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc1241959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37928/0 (socket says 172.21.15.90:37928) 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc122ffd640 1 -- 172.21.15.90:0/2289883116 learned_addr learned my addr 172.21.15.90:0/2289883116 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.125+0000 7fc1237fe640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 0x7fc124195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc122ffd640 1 -- 172.21.15.90:0/2289883116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 msgr2=0x7fc124195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc122ffd640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 0x7fc124195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc122ffd640 1 -- 172.21.15.90:0/2289883116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc118009560 con 0x7fc124100c80 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc1237fe640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 0x7fc124195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc122ffd640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc1241959b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc118004f60 tx=0x7fc118004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc118005460 con 0x7fc124100c80 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc1180055c0 con 0x7fc124100c80 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12419aad0 con 0x7fc124100c80 2024-09-13T15:33:01.129 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc11803c900 con 0x7fc124100c80 2024-09-13T15:33:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.126+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12419b020 con 0x7fc124100c80 2024-09-13T15:33:01.130 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.127+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1240ffe90 con 0x7fc124100c80 2024-09-13T15:33:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.127+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fc118003c80 con 0x7fc124100c80 2024-09-13T15:33:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.128+0000 7fc120ff9640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc10c06cc50 0x7fc10c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.128+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fc1180ada00 con 0x7fc124100c80 2024-09-13T15:33:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.128+0000 7fc1237fe640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc10c06cc50 0x7fc10c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.128+0000 7fc1237fe640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc10c06cc50 0x7fc10c06f110 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc124100ae0 tx=0x7fc114005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:01.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.132+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fc118077760 con 0x7fc124100c80 2024-09-13T15:33:01.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.309+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7fc12419b490 con 0x7fc124100c80 2024-09-13T15:33:01.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.309+0000 7fc120ff9640 1 -- 172.21.15.90:0/2289883116 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc11807ad80 con 0x7fc124100c80 2024-09-13T15:33:01.310 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:01.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc10c06cc50 msgr2=0x7fc10c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc10c06cc50 0x7fc10c06f110 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc124100ae0 tx=0x7fc114005f90 comp rx=0 tx=0).stop 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 msgr2=0x7fc1241959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc1241959b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fc118004f60 tx=0x7fc118004cd0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 shutdown_connections 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fc10c06cc50 0x7fc10c06f110 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fc124100c80 0x7fc1241959b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 --2- 172.21.15.90:0/2289883116 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1240ffa90 0x7fc124195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 >> 172.21.15.90:0/2289883116 conn(0x7fc1240fb1f0 msgr2=0x7fc124103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 shutdown_connections 2024-09-13T15:33:01.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.311+0000 7fc129e61640 1 -- 172.21.15.90:0/2289883116 wait complete. 2024-09-13T15:33:01.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T15:33:01.426 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 -- 172.21.15.90:0/3559481004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 msgr2=0x7f2878101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 --2- 172.21.15.90:0/3559481004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878101100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f286c009880 tx=0x7f286c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 -- 172.21.15.90:0/3559481004 shutdown_connections 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 --2- 172.21.15.90:0/3559481004 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878101100 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 --2- 172.21.15.90:0/3559481004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28780ffa90 0x7f28780ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 -- 172.21.15.90:0/3559481004 >> 172.21.15.90:0/3559481004 conn(0x7f28780fb1f0 msgr2=0x7f28780fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 -- 172.21.15.90:0/3559481004 shutdown_connections 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.423+0000 7f287e325640 1 -- 172.21.15.90:0/3559481004 wait complete. 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.424+0000 7f287e325640 1 Processor -- start 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.424+0000 7f287e325640 1 -- start start 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.424+0000 7f287e325640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28780ffa90 0x7f2878195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.427 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.424+0000 7f287e325640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.424+0000 7f287e325640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2878195ed0 con 0x7f2878100c80 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.424+0000 7f287e325640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2878196040 con 0x7f28780ffa90 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37958/0 (socket says 172.21.15.90:37958) 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 -- 172.21.15.90:0/3308070838 learned_addr learned my addr 172.21.15.90:0/3308070838 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f2877fff640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28780ffa90 0x7f2878195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 -- 172.21.15.90:0/3308070838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28780ffa90 msgr2=0x7f2878195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28780ffa90 0x7f2878195450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 -- 172.21.15.90:0/3308070838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286c009560 con 0x7f2878100c80 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28777fe640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878195990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f286c0099a0 tx=0x7f286c0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286c005560 con 0x7f2878100c80 2024-09-13T15:33:01.428 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f286c003a80 con 0x7f2878100c80 2024-09-13T15:33:01.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287819aab0 con 0x7f2878100c80 2024-09-13T15:33:01.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f286c03ca10 con 0x7f2878100c80 2024-09-13T15:33:01.429 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.425+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287819b000 con 0x7f2878100c80 2024-09-13T15:33:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.426+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28780ffe90 con 0x7f2878100c80 2024-09-13T15:33:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.427+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f286c003bf0 con 0x7f2878100c80 2024-09-13T15:33:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.427+0000 7f28757fa640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f286006cc60 0x7f286006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.427+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f286c0adb30 con 0x7f2878100c80 2024-09-13T15:33:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.427+0000 7f2877fff640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f286006cc60 0x7f286006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.430 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.428+0000 7f2877fff640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f286006cc60 0x7f286006f120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2868005fd0 tx=0x7f286800b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:01.432 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.431+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f286c077890 con 0x7f2878100c80 2024-09-13T15:33:01.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.641+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28781084c0 con 0x7f2878100c80 2024-09-13T15:33:01.643 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.642+0000 7f28757fa640 1 -- 172.21.15.90:0/3308070838 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4462 (secure 0 0 0) 0x7f286c07aeb0 con 0x7f2878100c80 2024-09-13T15:33:01.643 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:01.643 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T15:32:52.637540+0000","modified":"2024-09-13T15:33:00.767625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14514":{"gid":14514,"name":"foo.smithi090.zyijng","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/2333604411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":2333604411},{"type":"v1","addr":"172.21.15.90:6835","nonce":2333604411}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T15:32:58.742805+0000","modified":"2024-09-13T15:33:00.767625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24281":{"gid":24281,"name":"foo.smithi175.cvvnzp","rank":0,"incarnation":8,"state":"up:active","state_seq":2,"addr":"172.21.15.175:6833/2042693132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":2042693132},{"type":"v1","addr":"172.21.15.175:6833","nonce":2042693132}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T15:33:01.644 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f286006cc60 msgr2=0x7f286006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f286006cc60 0x7f286006f120 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2868005fd0 tx=0x7f286800b390 comp rx=0 tx=0).stop 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 msgr2=0x7f2878195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878195990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f286c0099a0 tx=0x7f286c0045e0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 shutdown_connections 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f286006cc60 0x7f286006f120 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.645 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f2878100c80 0x7f2878195990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 --2- 172.21.15.90:0/3308070838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28780ffa90 0x7f2878195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 >> 172.21.15.90:0/3308070838 conn(0x7f28780fb1f0 msgr2=0x7f2878103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 shutdown_connections 2024-09-13T15:33:01.646 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.644+0000 7f287e325640 1 -- 172.21.15.90:0/3308070838 wait complete. 2024-09-13T15:33:01.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:33:01.757 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 -- 172.21.15.90:0/4278883805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 msgr2=0x7fcbf01003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/4278883805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01003d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009880 tx=0x7fcbe802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 -- 172.21.15.90:0/4278883805 shutdown_connections 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/4278883805 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcbf00fec20 0x7fcbf00ff0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/4278883805 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01003d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 -- 172.21.15.90:0/4278883805 >> 172.21.15.90:0/4278883805 conn(0x7fcbf00f98a0 msgr2=0x7fcbf00fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 -- 172.21.15.90:0/4278883805 shutdown_connections 2024-09-13T15:33:01.758 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.756+0000 7fcbf4b25640 1 -- 172.21.15.90:0/4278883805 wait complete. 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbf4b25640 1 Processor -- start 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbf4b25640 1 -- start start 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbf4b25640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcbf00fec20 0x7fcbf01954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbf4b25640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbf4b25640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0195f20 con 0x7fcbf00fffd0 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbf4b25640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0196090 con 0x7fcbf00fec20 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbedd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbedd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37980/0 (socket says 172.21.15.90:37980) 2024-09-13T15:33:01.760 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbedd9b640 1 -- 172.21.15.90:0/3445238393 learned_addr learned my addr 172.21.15.90:0/3445238393 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:01.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbee59c640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcbf00fec20 0x7fcbf01954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbedd9b640 1 -- 172.21.15.90:0/3445238393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcbf00fec20 msgr2=0x7fcbf01954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.757+0000 7fcbedd9b640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcbf00fec20 0x7fcbf01954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.758+0000 7fcbedd9b640 1 -- 172.21.15.90:0/3445238393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe8009560 con 0x7fcbf00fffd0 2024-09-13T15:33:01.761 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.758+0000 7fcbedd9b640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01959e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400eaf0 tx=0x7fcbe400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:01.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.759+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe4003970 con 0x7fcbf00fffd0 2024-09-13T15:33:01.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.760+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf019ab60 con 0x7fcbf00fffd0 2024-09-13T15:33:01.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.760+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf019b110 con 0x7fcbf00fffd0 2024-09-13T15:33:01.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.760+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcbe4004020 con 0x7fcbf00fffd0 2024-09-13T15:33:01.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.760+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbe4010600 con 0x7fcbf00fffd0 2024-09-13T15:33:01.762 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.761+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf0107400 con 0x7fcbf00fffd0 2024-09-13T15:33:01.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.761+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fcbe4003ad0 con 0x7fcbf00fffd0 2024-09-13T15:33:01.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.762+0000 7fcbdf7fe640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcbd806cc50 0x7fcbd806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:01.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.762+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fcbe4014070 con 0x7fcbf00fffd0 2024-09-13T15:33:01.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.762+0000 7fcbee59c640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcbd806cc50 0x7fcbd806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:01.763 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.762+0000 7fcbee59c640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcbd806cc50 0x7fcbd806f110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009230 tx=0x7fcbe8004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:01.767 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.766+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fcbe4056d00 con 0x7fcbf00fffd0 2024-09-13T15:33:01.934 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.932+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbf0107590 con 0x7fcbf00fffd0 2024-09-13T15:33:01.935 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.934+0000 7fcbdf7fe640 1 -- 172.21.15.90:0/3445238393 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+21061 (secure 0 0 0) 0x7fcbe405a320 con 0x7fcbf00fffd0 2024-09-13T15:33:01.935 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:01.936 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:00.747847+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:33:01.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcbd806cc50 msgr2=0x7fcbd806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.937 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcbd806cc50 0x7fcbd806f110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009230 tx=0x7fcbe8004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 msgr2=0x7fcbf01959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01959e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400eaf0 tx=0x7fcbe400c2d0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 shutdown_connections 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fcbd806cc50 0x7fcbd806f110 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fcbf00fffd0 0x7fcbf01959e0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 --2- 172.21.15.90:0/3445238393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcbf00fec20 0x7fcbf01954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 >> 172.21.15.90:0/3445238393 conn(0x7fcbf00f98a0 msgr2=0x7fcbf00fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 shutdown_connections 2024-09-13T15:33:01.938 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:01.936+0000 7fcbf4b25640 1 -- 172.21.15.90:0/3445238393 wait complete. 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: pgmap v227: 129 pgs: 34 unknown, 25 creating+peering, 70 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s wr, 3 op/s 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] up:active 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] up:active 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1 {bar:0=foo.smithi175.cvvnzp=up:active,foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:01.947 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1061371609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:01.948 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2289883116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:01.948 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:01.948 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:01.948 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:01 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3308070838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-13T15:33:01.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:33:01.950 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:33:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: pgmap v227: 129 pgs: 34 unknown, 25 creating+peering, 70 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s wr, 3 op/s 2024-09-13T15:33:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: osdmap e69: 8 total, 8 up, 8 in 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] up:active 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] up:active 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1 {bar:0=foo.smithi175.cvvnzp=up:active,foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1061371609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2289883116' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:02.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:01 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3308070838' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- 172.21.15.90:0/4149449248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6014100ca0 msgr2=0x7f6014101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 --2- 172.21.15.90:0/4149449248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6014100ca0 0x7f6014101120 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6000009880 tx=0x7f600002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- 172.21.15.90:0/4149449248 shutdown_connections 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 --2- 172.21.15.90:0/4149449248 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6014100ca0 0x7f6014101120 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 --2- 172.21.15.90:0/4149449248 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60140ffab0 0x7f60140ffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- 172.21.15.90:0/4149449248 >> 172.21.15.90:0/4149449248 conn(0x7f60140fb250 msgr2=0x7f60140fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- 172.21.15.90:0/4149449248 shutdown_connections 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- 172.21.15.90:0/4149449248 wait complete. 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 Processor -- start 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- start start 2024-09-13T15:33:02.062 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 0x7f6014195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6014100ca0 0x7f6014195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6014195ec0 con 0x7f60140ffab0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.059+0000 7f601ab65640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6014196030 con 0x7f6014100ca0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 0x7f6014195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 0x7f6014195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:37994/0 (socket says 172.21.15.90:37994) 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 -- 172.21.15.90:0/2028420811 learned_addr learned my addr 172.21.15.90:0/2028420811 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6013fff640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6014100ca0 0x7f6014195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 -- 172.21.15.90:0/2028420811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6014100ca0 msgr2=0x7f6014195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6014100ca0 0x7f6014195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 -- 172.21.15.90:0/2028420811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6000009560 con 0x7f60140ffab0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6018901640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 0x7f6014195440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f600800d8d0 tx=0x7f600800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f600800bea0 con 0x7f60140ffab0 2024-09-13T15:33:02.063 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6008009c00 con 0x7f60140ffab0 2024-09-13T15:33:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601419ab00 con 0x7f60140ffab0 2024-09-13T15:33:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.060+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f600800f5d0 con 0x7f60140ffab0 2024-09-13T15:33:02.064 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.061+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601419b030 con 0x7f60140ffab0 2024-09-13T15:33:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.061+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60140ffeb0 con 0x7f60140ffab0 2024-09-13T15:33:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.062+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6008010460 con 0x7f60140ffab0 2024-09-13T15:33:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.062+0000 7f6011ffb640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f600406cc50 0x7f600406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.062+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f600808c930 con 0x7f60140ffab0 2024-09-13T15:33:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.062+0000 7f6013fff640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f600406cc50 0x7f600406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.063+0000 7f6013fff640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f600406cc50 0x7f600406f110 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6000000c00 tx=0x7f6000004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.067+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f60080565e0 con 0x7f60140ffab0 2024-09-13T15:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.234+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f601419b220 con 0x7f60140ffab0 2024-09-13T15:33:02.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.234+0000 7f6011ffb640 1 -- 172.21.15.90:0/2028420811 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6008059c00 con 0x7f60140ffab0 2024-09-13T15:33:02.235 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f600406cc50 msgr2=0x7f600406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f600406cc50 0x7f600406f110 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6000000c00 tx=0x7f6000004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 msgr2=0x7f6014195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 0x7f6014195440 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f600800d8d0 tx=0x7f600800dd90 comp rx=0 tx=0).stop 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 shutdown_connections 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f600406cc50 0x7f600406f110 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6014100ca0 0x7f6014195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 --2- 172.21.15.90:0/2028420811 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f60140ffab0 0x7f6014195440 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 >> 172.21.15.90:0/2028420811 conn(0x7f60140fb250 msgr2=0x7f6014103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 shutdown_connections 2024-09-13T15:33:02.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.236+0000 7f601ab65640 1 -- 172.21.15.90:0/2028420811 wait complete. 2024-09-13T15:33:02.248 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:33:02.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 -- 172.21.15.90:0/717930140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0fe9a0 msgr2=0x7f988c0feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 --2- 172.21.15.90:0/717930140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0fe9a0 0x7f988c0feda0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f987c009880 tx=0x7f987c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 -- 172.21.15.90:0/717930140 shutdown_connections 2024-09-13T15:33:02.352 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 --2- 172.21.15.90:0/717930140 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0ffb90 0x7f988c100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 --2- 172.21.15.90:0/717930140 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0fe9a0 0x7f988c0feda0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 -- 172.21.15.90:0/717930140 >> 172.21.15.90:0/717930140 conn(0x7f988c0fa100 msgr2=0x7f988c0fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 -- 172.21.15.90:0/717930140 shutdown_connections 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 -- 172.21.15.90:0/717930140 wait complete. 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.349+0000 7f9890ec9640 1 Processor -- start 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9890ec9640 1 -- start start 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9890ec9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0fe9a0 0x7f988c068b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9890ec9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 0x7f988c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9890ec9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c19a750 con 0x7f988c0ffb90 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9890ec9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c19a890 con 0x7f988c0fe9a0 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 0x7f988c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.353 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 0x7f988c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38010/0 (socket says 172.21.15.90:38010) 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 -- 172.21.15.90:0/32340057 learned_addr learned my addr 172.21.15.90:0/32340057 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f988a59c640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0fe9a0 0x7f988c068b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 -- 172.21.15.90:0/32340057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0fe9a0 msgr2=0x7f988c068b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0fe9a0 0x7f988c068b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 -- 172.21.15.90:0/32340057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c009560 con 0x7f988c0ffb90 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f988a59c640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0fe9a0 0x7f988c068b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9889d9b640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 0x7f988c19a210 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f987400e970 tx=0x7f987400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f987400cc90 con 0x7f988c0ffb90 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f987400cdf0 con 0x7f988c0ffb90 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c19ab40 con 0x7f988c0ffb90 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.350+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9874004e50 con 0x7f988c0ffb90 2024-09-13T15:33:02.354 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.351+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c19b120 con 0x7f988c0ffb90 2024-09-13T15:33:02.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.351+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c1073d0 con 0x7f988c0ffb90 2024-09-13T15:33:02.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.352+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f9874005020 con 0x7f988c0ffb90 2024-09-13T15:33:02.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.352+0000 7f98837fe640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f987806cc50 0x7f987806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.355 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.352+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f9874014070 con 0x7f988c0ffb90 2024-09-13T15:33:02.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.352+0000 7f988a59c640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f987806cc50 0x7f987806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.356 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.353+0000 7f988a59c640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f987806cc50 0x7f987806f110 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f987c009230 tx=0x7f987c004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.358 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.356+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f9874056ad0 con 0x7f988c0ffb90 2024-09-13T15:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.523+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7f988c19b370 con 0x7f988c0ffb90 2024-09-13T15:33:02.525 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.524+0000 7f98837fe640 1 -- 172.21.15.90:0/32340057 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f988c19b370 con 0x7f988c0ffb90 2024-09-13T15:33:02.525 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:02.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f987806cc50 msgr2=0x7f987806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f987806cc50 0x7f987806f110 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f987c009230 tx=0x7f987c004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:02.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 msgr2=0x7f988c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 0x7f988c19a210 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f987400e970 tx=0x7f987400ed70 comp rx=0 tx=0).stop 2024-09-13T15:33:02.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 shutdown_connections 2024-09-13T15:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f987806cc50 0x7f987806f110 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f988c0ffb90 0x7f988c19a210 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 --2- 172.21.15.90:0/32340057 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f988c0fe9a0 0x7f988c068b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 >> 172.21.15.90:0/32340057 conn(0x7f988c0fa100 msgr2=0x7f988c102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 shutdown_connections 2024-09-13T15:33:02.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.526+0000 7f9890ec9640 1 -- 172.21.15.90:0/32340057 wait complete. 2024-09-13T15:33:02.538 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.644+0000 7f5f8d86c640 1 -- 172.21.15.90:0/4193845260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 msgr2=0x7f5f880ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.644+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/4193845260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f880ffeb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009880 tx=0x7f5f7802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.644+0000 7f5f8d86c640 1 -- 172.21.15.90:0/4193845260 shutdown_connections 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.644+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/4193845260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 0x7f5f88101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.644+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/4193845260 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f880ffeb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.644+0000 7f5f8d86c640 1 -- 172.21.15.90:0/4193845260 >> 172.21.15.90:0/4193845260 conn(0x7f5f880fb250 msgr2=0x7f5f880fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 -- 172.21.15.90:0/4193845260 shutdown_connections 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 -- 172.21.15.90:0/4193845260 wait complete. 2024-09-13T15:33:02.648 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 Processor -- start 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 -- start start 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f88195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 0x7f5f88195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88195ea0 con 0x7f5f880ffab0 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f8d86c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88196010 con 0x7f5f88100ca0 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.645+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f88195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f86ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f88195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38032/0 (socket says 172.21.15.90:38032) 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f86ffd640 1 -- 172.21.15.90:0/1351144408 learned_addr learned my addr 172.21.15.90:0/1351144408 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f867fc640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 0x7f5f88195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f86ffd640 1 -- 172.21.15.90:0/1351144408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 msgr2=0x7f5f88195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.649 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f86ffd640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 0x7f5f88195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f86ffd640 1 -- 172.21.15.90:0/1351144408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c009490 con 0x7f5f880ffab0 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f867fc640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 0x7f5f88195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f86ffd640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f88195420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f78006010 tx=0x7f5f78004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f780053e0 con 0x7f5f880ffab0 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f78009560 con 0x7f5f880ffab0 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5f78005540 con 0x7f5f880ffab0 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f7803c810 con 0x7f5f880ffab0 2024-09-13T15:33:02.650 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.646+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8819ae40 con 0x7f5f880ffab0 2024-09-13T15:33:02.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.647+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f880ffeb0 con 0x7f5f880ffab0 2024-09-13T15:33:02.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.648+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f5f780035f0 con 0x7f5f880ffab0 2024-09-13T15:33:02.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.648+0000 7f5f8c86a640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5f7006cc50 0x7f5f7006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.648+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f5f780ada30 con 0x7f5f880ffab0 2024-09-13T15:33:02.651 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.648+0000 7f5f867fc640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5f7006cc50 0x7f5f7006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.652 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.649+0000 7f5f867fc640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5f7006cc50 0x7f5f7006f110 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5f88196930 tx=0x7f5f7c005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.654 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.653+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f5f78077790 con 0x7f5f880ffab0 2024-09-13T15:33:02.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.820+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7f5f8819b2e0 con 0x7f5f880ffab0 2024-09-13T15:33:02.822 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.820+0000 7f5f8c86a640 1 -- 172.21.15.90:0/1351144408 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f780056b0 con 0x7f5f880ffab0 2024-09-13T15:33:02.822 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5f7006cc50 msgr2=0x7f5f7006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5f7006cc50 0x7f5f7006f110 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5f88196930 tx=0x7f5f7c005f90 comp rx=0 tx=0).stop 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 msgr2=0x7f5f88195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f88195420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5f78006010 tx=0x7f5f78004c60 comp rx=0 tx=0).stop 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 shutdown_connections 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f5f7006cc50 0x7f5f7006f110 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f88100ca0 0x7f5f88195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 --2- 172.21.15.90:0/1351144408 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f5f880ffab0 0x7f5f88195420 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 >> 172.21.15.90:0/1351144408 conn(0x7f5f880fb250 msgr2=0x7f5f88103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.822+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 shutdown_connections 2024-09-13T15:33:02.825 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.823+0000 7f5f8d86c640 1 -- 172.21.15.90:0/1351144408 wait complete. 2024-09-13T15:33:02.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:02 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3445238393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:02.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:02 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2028420811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:02.833 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:02 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/32340057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:02.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 -- 172.21.15.90:0/2608567550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180ffb90 msgr2=0x7fd818100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 --2- 172.21.15.90:0/2608567550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180ffb90 0x7fd818100010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd800009880 tx=0x7fd80002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 -- 172.21.15.90:0/2608567550 shutdown_connections 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 --2- 172.21.15.90:0/2608567550 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180ffb90 0x7fd818100010 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 --2- 172.21.15.90:0/2608567550 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8180fe9a0 0x7fd8180feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 -- 172.21.15.90:0/2608567550 >> 172.21.15.90:0/2608567550 conn(0x7fd8180fa100 msgr2=0x7fd8180fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 -- 172.21.15.90:0/2608567550 shutdown_connections 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 -- 172.21.15.90:0/2608567550 wait complete. 2024-09-13T15:33:02.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.936+0000 7fd81d92c640 1 Processor -- start 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81d92c640 1 -- start start 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81d92c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 0x7fd818194360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81d92c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8180ffb90 0x7fd8181948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81d92c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818194de0 con 0x7fd8180fe9a0 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81d92c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd818194f50 con 0x7fd8180ffb90 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 0x7fd818194360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 0x7fd818194360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38040/0 (socket says 172.21.15.90:38040) 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 -- 172.21.15.90:0/1661293152 learned_addr learned my addr 172.21.15.90:0/1661293152 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd80ffff640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8180ffb90 0x7fd8181948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 -- 172.21.15.90:0/1661293152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8180ffb90 msgr2=0x7fd8181948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8180ffb90 0x7fd8181948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 -- 172.21.15.90:0/1661293152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800009560 con 0x7fd8180fe9a0 2024-09-13T15:33:02.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd816ffd640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 0x7fd818194360 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd80800d8d0 tx=0x7fd80800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80800bea0 con 0x7fd8180fe9a0 2024-09-13T15:33:02.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd818199a20 con 0x7fd8180fe9a0 2024-09-13T15:33:02.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd808009c00 con 0x7fd8180fe9a0 2024-09-13T15:33:02.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.937+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd80800f5d0 con 0x7fd8180fe9a0 2024-09-13T15:33:02.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.938+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd818199f50 con 0x7fd8180fe9a0 2024-09-13T15:33:02.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.938+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8180feda0 con 0x7fd8180fe9a0 2024-09-13T15:33:02.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.939+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd808010460 con 0x7fd8180fe9a0 2024-09-13T15:33:02.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.939+0000 7fd81c92a640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd80406cc50 0x7fd80406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:02.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.939+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fd80808c8e0 con 0x7fd8180fe9a0 2024-09-13T15:33:02.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.939+0000 7fd80ffff640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd80406cc50 0x7fd80406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:02.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.940+0000 7fd80ffff640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd80406cc50 0x7fd80406f110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd800000c00 tx=0x7fd800004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:02.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:02.944+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd808056590 con 0x7fd8180fe9a0 2024-09-13T15:33:03.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:02 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3445238393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:03.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:02 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2028420811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:03.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:02 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/32340057' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:03.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.119+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7fd81819a1a0 con 0x7fd8180fe9a0 2024-09-13T15:33:03.121 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.119+0000 7fd81c92a640 1 -- 172.21.15.90:0/1661293152 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd808054e60 con 0x7fd8180fe9a0 2024-09-13T15:33:03.121 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd80406cc50 msgr2=0x7fd80406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd80406cc50 0x7fd80406f110 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd800000c00 tx=0x7fd800004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 msgr2=0x7fd818194360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 0x7fd818194360 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd80800d8d0 tx=0x7fd80800dd90 comp rx=0 tx=0).stop 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 shutdown_connections 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd80406cc50 0x7fd80406f110 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8180ffb90 0x7fd8181948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 --2- 172.21.15.90:0/1661293152 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd8180fe9a0 0x7fd818194360 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 >> 172.21.15.90:0/1661293152 conn(0x7fd8180fa100 msgr2=0x7fd818102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 shutdown_connections 2024-09-13T15:33:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.121+0000 7fd81d92c640 1 -- 172.21.15.90:0/1661293152 wait complete. 2024-09-13T15:33:03.134 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:33:03.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 -- 172.21.15.90:0/3396147100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438072e90 msgr2=0x7f64380732f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 --2- 172.21.15.90:0/3396147100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438072e90 0x7f64380732f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6428009880 tx=0x7f642802a530 comp rx=0 tx=0).stop 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 -- 172.21.15.90:0/3396147100 shutdown_connections 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 --2- 172.21.15.90:0/3396147100 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438072e90 0x7f64380732f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 --2- 172.21.15.90:0/3396147100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6438074470 0x7f64380728c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 -- 172.21.15.90:0/3396147100 >> 172.21.15.90:0/3396147100 conn(0x7f64380fb1d0 msgr2=0x7f64380fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 -- 172.21.15.90:0/3396147100 shutdown_connections 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 -- 172.21.15.90:0/3396147100 wait complete. 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 Processor -- start 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 -- start start 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.237+0000 7f643fc53640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6438072e90 0x7f6438197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643fc53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 0x7f6438197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643fc53640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64381980f0 con 0x7f6438074470 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643fc53640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6438198260 con 0x7f6438072e90 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 0x7f6438197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 0x7f6438197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38060/0 (socket says 172.21.15.90:38060) 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 -- 172.21.15.90:0/3018436842 learned_addr learned my addr 172.21.15.90:0/3018436842 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d9ef640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6438072e90 0x7f6438197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 -- 172.21.15.90:0/3018436842 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6438072e90 msgr2=0x7f6438197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6438072e90 0x7f6438197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 -- 172.21.15.90:0/3018436842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6428009560 con 0x7f6438074470 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643d1ee640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 0x7f6438197bb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f64280099a0 tx=0x7f6428004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6428005460 con 0x7f6438074470 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643819ccd0 con 0x7f6438074470 2024-09-13T15:33:03.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f64280055c0 con 0x7f6438074470 2024-09-13T15:33:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f642803c940 con 0x7f6438074470 2024-09-13T15:33:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.238+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643819d220 con 0x7f6438074470 2024-09-13T15:33:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.240+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643810a6a0 con 0x7f6438074470 2024-09-13T15:33:03.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.240+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f642802ad50 con 0x7f6438074470 2024-09-13T15:33:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.241+0000 7f642effd640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f642406cca0 0x7f642406f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.241+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f64280ad8c0 con 0x7f6438074470 2024-09-13T15:33:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.241+0000 7f643d9ef640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f642406cca0 0x7f642406f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.242+0000 7f643d9ef640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f642406cca0 0x7f642406f160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6434009b60 tx=0x7f6434005f90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:03.247 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.246+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6428077560 con 0x7f6438074470 2024-09-13T15:33:03.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.413+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7f643819d4d0 con 0x7f6438074470 2024-09-13T15:33:03.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.414+0000 7f642effd640 1 -- 172.21.15.90:0/3018436842 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f643819d4d0 con 0x7f6438074470 2024-09-13T15:33:03.415 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f642406cca0 msgr2=0x7f642406f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f642406cca0 0x7f642406f160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6434009b60 tx=0x7f6434005f90 comp rx=0 tx=0).stop 2024-09-13T15:33:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 msgr2=0x7f6438197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 0x7f6438197bb0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f64280099a0 tx=0x7f6428004c60 comp rx=0 tx=0).stop 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 shutdown_connections 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f642406cca0 0x7f642406f160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6438074470 0x7f6438197bb0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 --2- 172.21.15.90:0/3018436842 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6438072e90 0x7f6438197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 >> 172.21.15.90:0/3018436842 conn(0x7f64380fb1d0 msgr2=0x7f64381058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 shutdown_connections 2024-09-13T15:33:03.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.416+0000 7f643fc53640 1 -- 172.21.15.90:0/3018436842 wait complete. 2024-09-13T15:33:03.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T15:33:03.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- 172.21.15.90:0/410030409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 msgr2=0x7fd3d40fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 --2- 172.21.15.90:0/410030409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d40fed90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009880 tx=0x7fd3c802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- 172.21.15.90:0/410030409 shutdown_connections 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 --2- 172.21.15.90:0/410030409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 0x7fd3d4100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 --2- 172.21.15.90:0/410030409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d40fed90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- 172.21.15.90:0/410030409 >> 172.21.15.90:0/410030409 conn(0x7fd3d40fa0f0 msgr2=0x7fd3d40fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- 172.21.15.90:0/410030409 shutdown_connections 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- 172.21.15.90:0/410030409 wait complete. 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 Processor -- start 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- start start 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d4067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 0x7fd3d419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d419a690 con 0x7fd3d40fe990 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3dae19640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d419a7d0 con 0x7fd3d40ffb80 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3d8bb5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d4067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.526+0000 7fd3d8bb5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d4067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38064/0 (socket says 172.21.15.90:38064) 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d8bb5640 1 -- 172.21.15.90:0/3846133791 learned_addr learned my addr 172.21.15.90:0/3846133791 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d3fff640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 0x7fd3d419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d8bb5640 1 -- 172.21.15.90:0/3846133791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 msgr2=0x7fd3d419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d8bb5640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 0x7fd3d419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d8bb5640 1 -- 172.21.15.90:0/3846133791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c8009560 con 0x7fd3d40fe990 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d3fff640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 0x7fd3d419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d8bb5640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d4067fc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8006010 tx=0x7fd3c8005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c8005600 con 0x7fd3d40fe990 2024-09-13T15:33:03.530 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd3c8003a80 con 0x7fd3d40fe990 2024-09-13T15:33:03.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d419aa20 con 0x7fd3d40fe990 2024-09-13T15:33:03.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd3c803ca90 con 0x7fd3d40fe990 2024-09-13T15:33:03.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.527+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d419afa0 con 0x7fd3d40fe990 2024-09-13T15:33:03.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.528+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d41073a0 con 0x7fd3d40fe990 2024-09-13T15:33:03.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.529+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd3c8003bf0 con 0x7fd3d40fe990 2024-09-13T15:33:03.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.529+0000 7fd3d1ffb640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd3c406cc50 0x7fd3c406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.530+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fd3c80adc90 con 0x7fd3d40fe990 2024-09-13T15:33:03.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.530+0000 7fd3d3fff640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd3c406cc50 0x7fd3c406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.532 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.530+0000 7fd3d3fff640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd3c406cc50 0x7fd3c406f110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4067d30 tx=0x7fd3c0005ec0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:03.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.534+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd3c8077940 con 0x7fd3d40fe990 2024-09-13T15:33:03.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.742+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3d4107560 con 0x7fd3d40fe990 2024-09-13T15:33:03.743 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.742+0000 7fd3d1ffb640 1 -- 172.21.15.90:0/3846133791 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4462 (secure 0 0 0) 0x7fd3c807af60 con 0x7fd3d40fe990 2024-09-13T15:33:03.744 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:03.744 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T15:32:52.637540+0000","modified":"2024-09-13T15:33:00.767625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14514":{"gid":14514,"name":"foo.smithi090.zyijng","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/2333604411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":2333604411},{"type":"v1","addr":"172.21.15.90:6835","nonce":2333604411}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T15:32:58.742805+0000","modified":"2024-09-13T15:33:00.767625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24281":{"gid":24281,"name":"foo.smithi175.cvvnzp","rank":0,"incarnation":8,"state":"up:active","state_seq":2,"addr":"172.21.15.175:6833/2042693132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":2042693132},{"type":"v1","addr":"172.21.15.175:6833","nonce":2042693132}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T15:33:03.744 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-09-13T15:33:03.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd3c406cc50 msgr2=0x7fd3c406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd3c406cc50 0x7fd3c406f110 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4067d30 tx=0x7fd3c0005ec0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 msgr2=0x7fd3d4067fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d4067fc0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8006010 tx=0x7fd3c8005000 comp rx=0 tx=0).stop 2024-09-13T15:33:03.746 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 shutdown_connections 2024-09-13T15:33:03.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd3c406cc50 0x7fd3c406f110 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3d40ffb80 0x7fd3d419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 --2- 172.21.15.90:0/3846133791 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3d40fe990 0x7fd3d4067fc0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 >> 172.21.15.90:0/3846133791 conn(0x7fd3d40fa0f0 msgr2=0x7fd3d4102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:03.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 shutdown_connections 2024-09-13T15:33:03.747 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.745+0000 7fd3dae19640 1 -- 172.21.15.90:0/3846133791 wait complete. 2024-09-13T15:33:03.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:33:03.864 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf037fe640 1 -- 172.21.15.90:0/965492747 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef80316d0 con 0x7faf040fdd50 2024-09-13T15:33:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 -- 172.21.15.90:0/965492747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 msgr2=0x7faf040fe150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/965492747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf040fe150 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7faef8009880 tx=0x7faef802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 -- 172.21.15.90:0/965492747 shutdown_connections 2024-09-13T15:33:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/965492747 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf04100d20 0x7faf041011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.868 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/965492747 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf040fe150 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 -- 172.21.15.90:0/965492747 >> 172.21.15.90:0/965492747 conn(0x7faf040f9870 msgr2=0x7faf040fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.863+0000 7faf0b0aa640 1 -- 172.21.15.90:0/965492747 shutdown_connections 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 -- 172.21.15.90:0/965492747 wait complete. 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 Processor -- start 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 -- start start 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf04195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf04100d20 0x7faf04195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04195ea0 con 0x7faf040fdd50 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf0b0aa640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04196010 con 0x7faf04100d20 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.864+0000 7faf08e46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf04195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf08e46640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf04195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38086/0 (socket says 172.21.15.90:38086) 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf08e46640 1 -- 172.21.15.90:0/1278672310 learned_addr learned my addr 172.21.15.90:0/1278672310 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:03.869 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf03fff640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf04100d20 0x7faf04195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf08e46640 1 -- 172.21.15.90:0/1278672310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf04100d20 msgr2=0x7faf04195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf08e46640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf04100d20 0x7faf04195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf08e46640 1 -- 172.21.15.90:0/1278672310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef8009560 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf08e46640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf04195420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faef8006010 tx=0x7faef8005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef80054e0 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0419aa80 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faef8005640 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faef803c990 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.865+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0419afd0 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.866+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf040fe150 con 0x7faf040fdd50 2024-09-13T15:33:03.870 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.867+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faef80035f0 con 0x7faf040fdd50 2024-09-13T15:33:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.867+0000 7faf01ffb640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faef406cc50 0x7faef406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.868+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7faef80adc30 con 0x7faf040fdd50 2024-09-13T15:33:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.868+0000 7faf03fff640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faef406cc50 0x7faef406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:03.871 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.868+0000 7faf03fff640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faef406cc50 0x7faef406f110 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faef0005fd0 tx=0x7faef0005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:03.872 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:03.871+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7faef80b4080 con 0x7faf040fdd50 2024-09-13T15:33:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: pgmap v229: 129 pgs: 18 unknown, 25 creating+peering, 86 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 10 op/s 2024-09-13T15:33:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1351144408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1661293152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3018436842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:03 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3846133791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-13T15:33:04.045 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.044+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf041084c0 con 0x7faf040fdd50 2024-09-13T15:33:04.047 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.046+0000 7faf01ffb640 1 -- 172.21.15.90:0/1278672310 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+21061 (secure 0 0 0) 0x7faef8077880 con 0x7faf040fdd50 2024-09-13T15:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:04.047 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:00.747847+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:33:04.050 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faef406cc50 msgr2=0x7faef406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faef406cc50 0x7faef406f110 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7faef0005fd0 tx=0x7faef0005850 comp rx=0 tx=0).stop 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 msgr2=0x7faf04195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf04195420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faef8006010 tx=0x7faef8005e50 comp rx=0 tx=0).stop 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 shutdown_connections 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faef406cc50 0x7faef406f110 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faf04100d20 0x7faf04195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 --2- 172.21.15.90:0/1278672310 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7faf040fdd50 0x7faf04195420 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 >> 172.21.15.90:0/1278672310 conn(0x7faf040f9870 msgr2=0x7faf04103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 shutdown_connections 2024-09-13T15:33:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.049+0000 7faf0b0aa640 1 -- 172.21.15.90:0/1278672310 wait complete. 2024-09-13T15:33:04.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: pgmap v229: 129 pgs: 18 unknown, 25 creating+peering, 86 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 10 op/s 2024-09-13T15:33:04.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1351144408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:04.061 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1661293152' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:04.062 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3018436842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:04.062 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:04.062 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:04.062 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:04.062 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:03 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3846133791' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-13T15:33:04.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T15:32:52.637540+0000', 'modified': '2024-09-13T15:33:00.767625+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 14514}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14514': {'gid': 14514, 'name': 'foo.smithi090.zyijng', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.90:6835/2333604411', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6834', 'nonce': 2333604411}, {'type': 'v1', 'addr': '172.21.15.90:6835', 'nonce': 2333604411}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'foo', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-09-13T15:33:04.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T15:33:04.064 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.foo.smithi090.zyijng status 2024-09-13T15:33:04.172 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 -- 172.21.15.90:0/2897622495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82981005c0 msgr2=0x7f82981029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 --2- 172.21.15.90:0/2897622495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82981005c0 0x7f82981029b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8288009880 tx=0x7f828802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 -- 172.21.15.90:0/2897622495 shutdown_connections 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 --2- 172.21.15.90:0/2897622495 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8298102ef0 0x7f82981052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 --2- 172.21.15.90:0/2897622495 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82981005c0 0x7f82981029b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 -- 172.21.15.90:0/2897622495 >> 172.21.15.90:0/2897622495 conn(0x7f82980fa100 msgr2=0x7f82980fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 -- 172.21.15.90:0/2897622495 shutdown_connections 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.171+0000 7f829d627640 1 -- 172.21.15.90:0/2897622495 wait complete. 2024-09-13T15:33:04.173 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f829d627640 1 Processor -- start 2024-09-13T15:33:04.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f829d627640 1 -- start start 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f829d627640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82981005c0 0x7f8298195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f829d627640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 0x7f8298195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f829d627640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8298195e90 con 0x7f8298102ef0 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f829d627640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8298196000 con 0x7f82981005c0 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 0x7f8298195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 0x7f8298195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38108/0 (socket says 172.21.15.90:38108) 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 -- 172.21.15.90:0/3960745616 learned_addr learned my addr 172.21.15.90:0/3960745616 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 -- 172.21.15.90:0/3960745616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82981005c0 msgr2=0x7f8298195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82981005c0 0x7f8298195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 -- 172.21.15.90:0/3960745616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8288009560 con 0x7f8298102ef0 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.172+0000 7f82967fc640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 0x7f8298195950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f828c00d6c0 tx=0x7f828c00db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.173+0000 7f827ffff640 1 -- 172.21.15.90:0/3960745616 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c0051f0 con 0x7f8298102ef0 2024-09-13T15:33:04.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.173+0000 7f827ffff640 1 -- 172.21.15.90:0/3960745616 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f828c009820 con 0x7f8298102ef0 2024-09-13T15:33:04.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.173+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829819aad0 con 0x7f8298102ef0 2024-09-13T15:33:04.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.173+0000 7f827ffff640 1 -- 172.21.15.90:0/3960745616 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c004ca0 con 0x7f8298102ef0 2024-09-13T15:33:04.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.173+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829819b080 con 0x7f8298102ef0 2024-09-13T15:33:04.177 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.174+0000 7f827ffff640 1 -- 172.21.15.90:0/3960745616 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f828c009990 con 0x7f8298102ef0 2024-09-13T15:33:04.182 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.175+0000 7f827ffff640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f828006cc50 0x7f828006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.175+0000 7f827ffff640 1 -- 172.21.15.90:0/3960745616 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f828c08c330 con 0x7f8298102ef0 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.175+0000 7f8296ffd640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f828006cc50 0x7f828006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.175+0000 7f82a0423740 1 Processor -- start 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.175+0000 7f8296ffd640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f828006cc50 0x7f828006f110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f828802a9c0 tx=0x7f8288000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.175+0000 7f82a0423740 1 -- start start 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f82a0423740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7581b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f82a0423740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd75d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f82a0423740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd6c1c60 con 0x5581bd757db0 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f82a0423740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd6bf3f0 con 0x5581bd7586f0 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38110/0 (socket says 172.21.15.90:38110) 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 -- 172.21.15.90:0/840381271 learned_addr learned my addr 172.21.15.90:0/840381271 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.183 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f82967fc640 1 --2- 172.21.15.90:0/840381271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd75d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 -- 172.21.15.90:0/840381271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 msgr2=0x5581bd75d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 --2- 172.21.15.90:0/840381271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd75d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 -- 172.21.15.90:0/840381271 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581bd67cf70 con 0x5581bd757db0 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f8296ffd640 1 --2- 172.21.15.90:0/840381271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7581b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8288004540 tx=0x7f828803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625abc99dcbebe00 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f827d7fa640 1 -- 172.21.15.90:0/840381271 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8288004c80 con 0x5581bd757db0 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f827d7fa640 1 -- 172.21.15.90:0/840381271 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8288005490 con 0x5581bd757db0 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.176+0000 7f827d7fa640 1 -- 172.21.15.90:0/840381271 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8288003e60 con 0x5581bd757db0 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 -- 172.21.15.90:0/840381271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 msgr2=0x5581bd7581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 --2- 172.21.15.90:0/840381271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7581b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8288004540 tx=0x7f828803c000 comp rx=0 tx=0).stop 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 -- 172.21.15.90:0/840381271 shutdown_connections 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 --2- 172.21.15.90:0/840381271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd75d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 --2- 172.21.15.90:0/840381271 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7581b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.184 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 -- 172.21.15.90:0/840381271 >> 172.21.15.90:0/840381271 conn(0x5581bd753460 msgr2=0x5581bd753860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 -- 172.21.15.90:0/840381271 shutdown_connections 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 -- 172.21.15.90:0/840381271 wait complete. 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 -- start start 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.177+0000 7f82a0423740 1 Processor -- start 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd7616f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd6bace0 con 0x5581bd757db0 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd679300 con 0x5581bd7586f0 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 10 client.0 _ll_register_callbacks cb 0x5581bd644e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7611b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38120/0 (socket says 172.21.15.90:38120) 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 -- 172.21.15.90:0/1737392242 learned_addr learned my addr 172.21.15.90:0/1737392242 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.185 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82967fc640 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd7616f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 -- 172.21.15.90:0/1737392242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 msgr2=0x5581bd7616f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd7616f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 -- 172.21.15.90:0/1737392242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8298105a10 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f8296ffd640 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7611b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8288004650 tx=0x7f8288004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb14f7c26ef2594 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f827aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f827aff5640 1 -- 172.21.15.90:0/1737392242 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8288003ee0 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f827aff5640 1 -- 172.21.15.90:0/1737392242 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f82880051d0 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581bd6e37c0 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f827aff5640 1 -- 172.21.15.90:0/1737392242 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f828805c470 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 10 client.14586 fetch_fsmap learned FSMap version 9 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.178+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=3+}) v3 -- 0x5581bd6b9360 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f827aff5640 1 -- 172.21.15.90:0/1737392242 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5581bd6e37c0 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f827aff5640 1 -- 172.21.15.90:0/1737392242 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) v1 ==== 2620+0+0 (secure 0 0 0) 0x7f8288032470 con 0x5581bd757db0 2024-09-13T15:33:04.186 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82a0423740 10 client.14586 fetch_fsmap finished waiting for FSMap version 9 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82a0423740 10 client.14586 resolve_mds: resolved name 'foo.smithi090.zyijng' to daemon mds.foo.smithi090.zyijng 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82a0423740 20 client.14586 populate_metadata read hostname 'smithi090' 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82a0423740 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd76aae0 0x5581bd76ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82a0423740 4 client.14586 mds_command: new command op to 14514 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 --> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581bd6aeda0 con 0x5581bd76aae0 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82967fc640 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd76aae0 0x5581bd76ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f827aff5640 10 client.14586 ms_handle_connect on v2:172.21.15.90:6834/2333604411 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.179+0000 7f82967fc640 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd76aae0 0x5581bd76ced0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70f2176f07fa4f2f server_cookie=20a41cddea36554e in_seq=0 out_seq=0 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f827aff5640 1 -- 172.21.15.90:0/1737392242 <== mds.0 v2:172.21.15.90:6834/2333604411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5581bd6aeda0 con 0x5581bd76aae0 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f827aff5640 10 client.14586 handle_command_reply: tid=0 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 client.14586 shutdown 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 msgr2=0x5581bd7611b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7611b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8288004650 tx=0x7f8288004d60 comp rx=0 tx=0).stop 2024-09-13T15:33:04.187 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 shutdown_connections 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd76aae0 0x5581bd76ced0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd7586f0 0x5581bd7616f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 --2- 172.21.15.90:0/1737392242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd757db0 0x5581bd7611b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 >> 172.21.15.90:0/1737392242 conn(0x5581bd753460 msgr2=0x5581bd7564b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f827aff5640 0 client.14586 ms_handle_reset on v2:172.21.15.90:6834/2333604411 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 shutdown_connections 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 1 -- 172.21.15.90:0/1737392242 wait complete. 2024-09-13T15:33:04.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.180+0000 7f82a0423740 20 client.14586 trim_cache size 0 max 16384 2024-09-13T15:33:04.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.194+0000 7f82a0423740 1 Processor -- start 2024-09-13T15:33:04.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.194+0000 7f82a0423740 1 -- start start 2024-09-13T15:33:04.200 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.194+0000 7f82a0423740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd75d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.194+0000 7f82a0423740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82a0423740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd678fa0 con 0x5581bd7586f0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82a0423740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd63e5b0 con 0x5581bd757db0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38128/0 (socket says 172.21.15.90:38128) 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 -- 172.21.15.90:0/1879201828 learned_addr learned my addr 172.21.15.90:0/1879201828 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f8296ffd640 1 --2- 172.21.15.90:0/1879201828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd75d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 -- 172.21.15.90:0/1879201828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 msgr2=0x5581bd75d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 --2- 172.21.15.90:0/1879201828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd75d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 -- 172.21.15.90:0/1879201828 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581bd65b6e0 con 0x5581bd7586f0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f82967fc640 1 --2- 172.21.15.90:0/1879201828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77ad60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f828c059a20 tx=0x7f828c08a190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=347cf9319fa515f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f827dffb640 1 -- 172.21.15.90:0/1879201828 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c057e60 con 0x5581bd7586f0 2024-09-13T15:33:04.201 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f827dffb640 1 -- 172.21.15.90:0/1879201828 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f828c08a3d0 con 0x5581bd7586f0 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.195+0000 7f827dffb640 1 -- 172.21.15.90:0/1879201828 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c005350 con 0x5581bd7586f0 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- 172.21.15.90:0/1879201828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 msgr2=0x5581bd77ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 --2- 172.21.15.90:0/1879201828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77ad60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f828c059a20 tx=0x7f828c08a190 comp rx=0 tx=0).stop 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- 172.21.15.90:0/1879201828 shutdown_connections 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 --2- 172.21.15.90:0/1879201828 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77ad60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 --2- 172.21.15.90:0/1879201828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd75d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- 172.21.15.90:0/1879201828 >> 172.21.15.90:0/1879201828 conn(0x5581bd753460 msgr2=0x5581bd75c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- 172.21.15.90:0/1879201828 shutdown_connections 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- 172.21.15.90:0/1879201828 wait complete. 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- start start 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 Processor -- start 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd77a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.202 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd77b350 con 0x5581bd7586f0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581bd77b490 con 0x5581bd757db0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82a0423740 10 client.0 _ll_register_callbacks cb 0x7f8288009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82967fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:38130/0 (socket says 172.21.15.90:38130) 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f82967fc640 1 -- 172.21.15.90:0/818954883 learned_addr learned my addr 172.21.15.90:0/818954883 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.196+0000 7f8296ffd640 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd77a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82967fc640 1 -- 172.21.15.90:0/818954883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 msgr2=0x5581bd77a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82967fc640 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd77a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82967fc640 1 -- 172.21.15.90:0/818954883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829819b080 con 0x5581bd7586f0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f8296ffd640 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd77a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82967fc640 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77adc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f828c005d00 tx=0x7f828c05a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9069b031d7b4c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f827aff5640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-09-13T15:33:04.203 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f827aff5640 1 -- 172.21.15.90:0/818954883 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f828c067070 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f827aff5640 1 -- 172.21.15.90:0/818954883 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f828c06db60 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581bd74f100 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f827aff5640 1 -- 172.21.15.90:0/818954883 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f828c056600 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 10 client.14590 fetch_fsmap learned FSMap version 9 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=3+}) v3 -- 0x5581bd680df0 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f827aff5640 1 -- 172.21.15.90:0/818954883 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5581bd74f100 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f827aff5640 1 -- 172.21.15.90:0/818954883 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) v1 ==== 2620+0+0 (secure 0 0 0) 0x7f828c06a020 con 0x5581bd7586f0 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 10 client.14590 fetch_fsmap finished waiting for FSMap version 9 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 10 client.14590 resolve_mds: resolved name 'foo.smithi090.zyijng' to daemon mds.foo.smithi090.zyijng 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 20 client.14590 populate_metadata read hostname 'smithi090' 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd7c1cd0 0x5581bd7c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 4 client.14590 mds_command: new command op to 14514 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T15:33:04.204 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 --> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f8288005490 con 0x5581bd7c1cd0 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.197+0000 7f8296ffd640 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd7c1cd0 0x5581bd7c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f827aff5640 10 client.14590 ms_handle_connect on v2:172.21.15.90:6834/2333604411 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f8296ffd640 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd7c1cd0 0x5581bd7c40e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d943d4198327cb38 server_cookie=d5f4136b3e810b87 in_seq=0 out_seq=0 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f827aff5640 1 -- 172.21.15.90:0/818954883 <== mds.0 v2:172.21.15.90:6834/2333604411 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x7f8288005490 con 0x5581bd7c1cd0 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f827aff5640 10 client.14590 handle_command_reply: tid=0 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 client.14590 shutdown 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 msgr2=0x5581bd77adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77adc0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f828c005d00 tx=0x7f828c05a560 comp rx=0 tx=0).stop 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 shutdown_connections 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:6834/2333604411,v1:172.21.15.90:6835/2333604411] conn(0x5581bd7c1cd0 0x5581bd7c40e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581bd7586f0 0x5581bd77adc0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f827aff5640 0 client.14590 ms_handle_reset on v2:172.21.15.90:6834/2333604411 2024-09-13T15:33:04.205 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 --2- 172.21.15.90:0/818954883 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581bd757db0 0x5581bd77a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 >> 172.21.15.90:0/818954883 conn(0x5581bd753460 msgr2=0x5581bd760f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 shutdown_connections 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 1 -- 172.21.15.90:0/818954883 wait complete. 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f82a0423740 20 client.14590 trim_cache size 0 max 16384 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "41124e1c-71e4-11ef-bceb-c7b262605968", 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 14514, 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-09-13T15:33:04.206 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "foo", 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 7.5417565309999999, 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 9, 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 69, 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 69, 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 7.6065098329999996 2024-09-13T15:33:04.207 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f828006cc50 msgr2=0x7f828006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.198+0000 7f829d627640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f828006cc50 0x7f828006f110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f828802a9c0 tx=0x7f8288000bc0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 msgr2=0x7f8298195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 0x7f8298195950 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f828c00d6c0 tx=0x7f828c00db80 comp rx=0 tx=0).stop 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 shutdown_connections 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f828006cc50 0x7f828006f110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8298102ef0 0x7f8298195950 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 --2- 172.21.15.90:0/3960745616 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82981005c0 0x7f8298195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 >> 172.21.15.90:0/3960745616 conn(0x7f82980fa100 msgr2=0x7f82980fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 shutdown_connections 2024-09-13T15:33:04.208 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.199+0000 7f829d627640 1 -- 172.21.15.90:0/3960745616 wait complete. 2024-09-13T15:33:04.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:33:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 -- 172.21.15.90:0/662391110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 msgr2=0x7f6d8c0feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 --2- 172.21.15.90:0/662391110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c0feda0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009880 tx=0x7f6d8802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 -- 172.21.15.90:0/662391110 shutdown_connections 2024-09-13T15:33:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 --2- 172.21.15.90:0/662391110 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d8c0ffb90 0x7f6d8c100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 --2- 172.21.15.90:0/662391110 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c0feda0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.310 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 -- 172.21.15.90:0/662391110 >> 172.21.15.90:0/662391110 conn(0x7f6d8c0fa100 msgr2=0x7f6d8c0fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 -- 172.21.15.90:0/662391110 shutdown_connections 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.307+0000 7f6d94212640 1 -- 172.21.15.90:0/662391110 wait complete. 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d94212640 1 Processor -- start 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d94212640 1 -- start start 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d94212640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c068ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d94212640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d8c0ffb90 0x7f6d8c19c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d94212640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c195f90 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d94212640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c19caf0 con 0x7f6d8c0ffb90 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c068ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c068ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55614/0 (socket says 172.21.15.90:55614) 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 -- 172.21.15.90:0/490298573 learned_addr learned my addr 172.21.15.90:0/490298573 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 -- 172.21.15.90:0/490298573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d8c0ffb90 msgr2=0x7f6d8c19c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.311 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d8c0ffb90 0x7f6d8c19c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 -- 172.21.15.90:0/490298573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d88009560 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.308+0000 7f6d91fae640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c068ba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6d88006010 tx=0x7f6d88004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.309+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d880055b0 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.309+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6d88003a80 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.309+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c19cc90 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.309+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6d8803c9e0 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.312 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.309+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c19d1b0 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.309+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c1073b0 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.310+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6d88003bf0 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.310+0000 7f6d82ffd640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6d7806cc50 0x7f6d7806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.310+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f6d880adcb0 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.310+0000 7f6d917ad640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6d7806cc50 0x7f6d7806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.313 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.311+0000 7f6d917ad640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6d7806cc50 0x7f6d7806f110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c068910 tx=0x7f6d7c005bf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.315 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.315+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6d88077960 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.487 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.486+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d8c107540 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.489 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:04.489 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.487+0000 7f6d82ffd640 1 -- 172.21.15.90:0/490298573 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+21061 (secure 0 0 0) 0x7f6d8807af80 con 0x7f6d8c0fe9a0 2024-09-13T15:33:04.490 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:00.747847+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6d7806cc50 msgr2=0x7f6d7806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6d7806cc50 0x7f6d7806f110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c068910 tx=0x7f6d7c005bf0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 msgr2=0x7f6d8c068ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c068ba0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6d88006010 tx=0x7f6d88004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 shutdown_connections 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6d7806cc50 0x7f6d7806f110 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d8c0ffb90 0x7f6d8c19c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.491 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 --2- 172.21.15.90:0/490298573 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6d8c0fe9a0 0x7f6d8c068ba0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 >> 172.21.15.90:0/490298573 conn(0x7f6d8c0fa100 msgr2=0x7f6d8c102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 shutdown_connections 2024-09-13T15:33:04.492 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.490+0000 7f6d94212640 1 -- 172.21.15.90:0/490298573 wait complete. 2024-09-13T15:33:04.502 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:33:04.502 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 -- 172.21.15.90:0/988354705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 msgr2=0x7f771c0fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 --2- 172.21.15.90:0/988354705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c0fed70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f770c009880 tx=0x7f770c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 -- 172.21.15.90:0/988354705 shutdown_connections 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 --2- 172.21.15.90:0/988354705 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f771c0ffb60 0x7f771c0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 --2- 172.21.15.90:0/988354705 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c0fed70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 -- 172.21.15.90:0/988354705 >> 172.21.15.90:0/988354705 conn(0x7f771c0fa0f0 msgr2=0x7f771c0fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 -- 172.21.15.90:0/988354705 shutdown_connections 2024-09-13T15:33:04.611 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.608+0000 7f7721061640 1 -- 172.21.15.90:0/988354705 wait complete. 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 Processor -- start 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 -- start start 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f771c0ffb60 0x7f771c19a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c19a690 con 0x7f771c0fe970 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771c19a7d0 con 0x7f771c0ffb60 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55630/0 (socket says 172.21.15.90:55630) 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 -- 172.21.15.90:0/1504602213 learned_addr learned my addr 172.21.15.90:0/1504602213 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 -- 172.21.15.90:0/1504602213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f771c0ffb60 msgr2=0x7f771c19a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f771c0ffb60 0x7f771c19a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.612 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 -- 172.21.15.90:0/1504602213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c009560 con 0x7f771c0fe970 2024-09-13T15:33:04.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f771ad9d640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c067fc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f770c006010 tx=0x7f770c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f770c005600 con 0x7f771c0fe970 2024-09-13T15:33:04.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f770c003a80 con 0x7f771c0fe970 2024-09-13T15:33:04.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c19aa20 con 0x7f771c0fe970 2024-09-13T15:33:04.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f770c03ca90 con 0x7f771c0fe970 2024-09-13T15:33:04.613 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.609+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c19afa0 con 0x7f771c0fe970 2024-09-13T15:33:04.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.611+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c1073a0 con 0x7f771c0fe970 2024-09-13T15:33:04.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.611+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f770c03cbf0 con 0x7f771c0fe970 2024-09-13T15:33:04.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.611+0000 7f7718d99640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f770806cc50 0x7f770806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.611+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f770c0adbe0 con 0x7f771c0fe970 2024-09-13T15:33:04.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.611+0000 7f7713fff640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f770806cc50 0x7f770806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.614 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.612+0000 7f7713fff640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f770806cc50 0x7f770806f110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f771c067d30 tx=0x7f7704005bf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.616 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.615+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f770c077910 con 0x7f771c0fe970 2024-09-13T15:33:04.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.780+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f771c19b480 con 0x7f771c0fe970 2024-09-13T15:33:04.782 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.781+0000 7f7718d99640 1 -- 172.21.15.90:0/1504602213 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f770c040020 con 0x7f771c0fe970 2024-09-13T15:33:04.782 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:33:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f770806cc50 msgr2=0x7f770806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f770806cc50 0x7f770806f110 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f771c067d30 tx=0x7f7704005bf0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.784 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 msgr2=0x7f771c067fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c067fc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f770c006010 tx=0x7f770c005000 comp rx=0 tx=0).stop 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 shutdown_connections 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f770806cc50 0x7f770806f110 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f771c0ffb60 0x7f771c19a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 --2- 172.21.15.90:0/1504602213 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f771c0fe970 0x7f771c067fc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 >> 172.21.15.90:0/1504602213 conn(0x7f771c0fa0f0 msgr2=0x7f771c102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 shutdown_connections 2024-09-13T15:33:04.785 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.783+0000 7f7721061640 1 -- 172.21.15.90:0/1504602213 wait complete. 2024-09-13T15:33:04.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:33:04.899 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 -- 172.21.15.90:0/3182013212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 msgr2=0x7fa334100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/3182013212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa334100010 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa324009880 tx=0x7fa32402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 -- 172.21.15.90:0/3182013212 shutdown_connections 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/3182013212 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa334100010 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/3182013212 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3340fe9a0 0x7fa3340feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 -- 172.21.15.90:0/3182013212 >> 172.21.15.90:0/3182013212 conn(0x7fa3340fa100 msgr2=0x7fa3340fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 -- 172.21.15.90:0/3182013212 shutdown_connections 2024-09-13T15:33:04.900 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 -- 172.21.15.90:0/3182013212 wait complete. 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.896+0000 7fa33bf3a640 1 Processor -- start 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa33bf3a640 1 -- start start 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa33bf3a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3340fe9a0 0x7fa334068b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa33bf3a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa33419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa33bf3a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33419a6f0 con 0x7fa3340ffb90 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa33bf3a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa33419a830 con 0x7fa3340fe9a0 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa33419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa33419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55646/0 (socket says 172.21.15.90:55646) 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 -- 172.21.15.90:0/461384083 learned_addr learned my addr 172.21.15.90:0/461384083 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa339cd6640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3340fe9a0 0x7fa334068b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.901 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 -- 172.21.15.90:0/461384083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3340fe9a0 msgr2=0x7fa334068b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3340fe9a0 0x7fa334068b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 -- 172.21.15.90:0/461384083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa324009560 con 0x7fa3340ffb90 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa3394d5640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa33419a1b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa324004f60 tx=0x7fa324004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3240054e0 con 0x7fa3340ffb90 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.897+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33419aa80 con 0x7fa3340ffb90 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.898+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa324005640 con 0x7fa3340ffb90 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.898+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa32403c960 con 0x7fa3340ffb90 2024-09-13T15:33:04.902 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.898+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33419b000 con 0x7fa3340ffb90 2024-09-13T15:33:04.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.899+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3341073b0 con 0x7fa3340ffb90 2024-09-13T15:33:04.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.899+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa324003cf0 con 0x7fa3340ffb90 2024-09-13T15:33:04.903 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.900+0000 7fa32affd640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa32006cc50 0x7fa32006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:04.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.900+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7fa3240adba0 con 0x7fa3340ffb90 2024-09-13T15:33:04.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.900+0000 7fa339cd6640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa32006cc50 0x7fa32006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:04.904 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.900+0000 7fa339cd6640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa32006cc50 0x7fa32006f110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa330005fd0 tx=0x7fa330005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:04.905 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:04.904+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa324077850 con 0x7fa3340ffb90 2024-09-13T15:33:05.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:05.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:05.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.snmuxm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.snmuxm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: Deploying daemon mds.bar.smithi090.snmuxm on smithi090 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1278672310' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/490298573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:05.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:04 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1504602213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.070+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7fa33419b4d0 con 0x7fa3340ffb90 2024-09-13T15:33:05.072 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.071+0000 7fa32affd640 1 -- 172.21.15.90:0/461384083 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa32407ae70 con 0x7fa3340ffb90 2024-09-13T15:33:05.072 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:05.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa32006cc50 msgr2=0x7fa32006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa32006cc50 0x7fa32006f110 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa330005fd0 tx=0x7fa330005850 comp rx=0 tx=0).stop 2024-09-13T15:33:05.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 msgr2=0x7fa33419a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.074 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa33419a1b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa324004f60 tx=0x7fa324004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 shutdown_connections 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa32006cc50 0x7fa32006f110 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa3340ffb90 0x7fa33419a1b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 --2- 172.21.15.90:0/461384083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3340fe9a0 0x7fa334068b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 >> 172.21.15.90:0/461384083 conn(0x7fa3340fa100 msgr2=0x7fa334102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 shutdown_connections 2024-09-13T15:33:05.075 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.073+0000 7fa33bf3a640 1 -- 172.21.15.90:0/461384083 wait complete. 2024-09-13T15:33:05.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:33:05.188 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 -- 172.21.15.90:0/2526335242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 msgr2=0x7f12600fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 --2- 172.21.15.90:0/2526335242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f12600fffe0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1254009880 tx=0x7f125402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 -- 172.21.15.90:0/2526335242 shutdown_connections 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 --2- 172.21.15.90:0/2526335242 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f12600fffe0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 --2- 172.21.15.90:0/2526335242 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 0x7f12600fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 -- 172.21.15.90:0/2526335242 >> 172.21.15.90:0/2526335242 conn(0x7f12600fa0f0 msgr2=0x7f12600fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 -- 172.21.15.90:0/2526335242 shutdown_connections 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.185+0000 7f1265745640 1 -- 172.21.15.90:0/2526335242 wait complete. 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f1265745640 1 Processor -- start 2024-09-13T15:33:05.189 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f1265745640 1 -- start start 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f1265745640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 0x7f1260067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f1265745640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f126019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f1265745640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126019a780 con 0x7f12600ffb60 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f1265745640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126019a8c0 con 0x7f12600fe970 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f125e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f126019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f125e7fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f126019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55664/0 (socket says 172.21.15.90:55664) 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.186+0000 7f125e7fc640 1 -- 172.21.15.90:0/2560699658 learned_addr learned my addr 172.21.15.90:0/2560699658 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f125effd640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 0x7f1260067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f125e7fc640 1 -- 172.21.15.90:0/2560699658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 msgr2=0x7f1260067fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f125e7fc640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 0x7f1260067fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f125e7fc640 1 -- 172.21.15.90:0/2560699658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1254009560 con 0x7f12600ffb60 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f125effd640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 0x7f1260067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:33:05.190 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f125e7fc640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f126019a240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1254005940 tx=0x7f1254005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:05.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1254005630 con 0x7f12600ffb60 2024-09-13T15:33:05.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1254003a80 con 0x7f12600ffb60 2024-09-13T15:33:05.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126019ab10 con 0x7f12600ffb60 2024-09-13T15:33:05.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f125403cb20 con 0x7f12600ffb60 2024-09-13T15:33:05.191 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.187+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126019b060 con 0x7f12600ffb60 2024-09-13T15:33:05.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.188+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1260107420 con 0x7f12600ffb60 2024-09-13T15:33:05.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.188+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f1254003bf0 con 0x7f12600ffb60 2024-09-13T15:33:05.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.189+0000 7f1247fff640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f124806cc50 0x7f124806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.189+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f12540ade00 con 0x7f12600ffb60 2024-09-13T15:33:05.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.189+0000 7f125effd640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f124806cc50 0x7f124806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.192 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.189+0000 7f125effd640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f124806cc50 0x7f124806f110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12600ff9c0 tx=0x7f12500073c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:05.194 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.193+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f1254077ab0 con 0x7f12600ffb60 2024-09-13T15:33:05.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:05.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.snmuxm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi090.snmuxm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: Deploying daemon mds.bar.smithi090.snmuxm on smithi090 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1278672310' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/490298573' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:05.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:04 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1504602213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:05.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.358+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7f126019b4b0 con 0x7f12600ffb60 2024-09-13T15:33:05.359 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.358+0000 7f1247fff640 1 -- 172.21.15.90:0/2560699658 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f125407b0d0 con 0x7f12600ffb60 2024-09-13T15:33:05.359 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:05.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f124806cc50 msgr2=0x7f124806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.361 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f124806cc50 0x7f124806f110 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f12600ff9c0 tx=0x7f12500073c0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 msgr2=0x7f126019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f126019a240 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1254005940 tx=0x7f1254005230 comp rx=0 tx=0).stop 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 shutdown_connections 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f124806cc50 0x7f124806f110 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f12600ffb60 0x7f126019a240 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 --2- 172.21.15.90:0/2560699658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12600fe970 0x7f1260067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 >> 172.21.15.90:0/2560699658 conn(0x7f12600fa0f0 msgr2=0x7f1260102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 shutdown_connections 2024-09-13T15:33:05.362 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.360+0000 7f1265745640 1 -- 172.21.15.90:0/2560699658 wait complete. 2024-09-13T15:33:05.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:33:05.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 -- 172.21.15.90:0/4056265761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800fe970 msgr2=0x7f6f800fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 --2- 172.21.15.90:0/4056265761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800fe970 0x7f6f800fed70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6f70009880 tx=0x7f6f7002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 -- 172.21.15.90:0/4056265761 shutdown_connections 2024-09-13T15:33:05.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 --2- 172.21.15.90:0/4056265761 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f800ffb60 0x7f6f800fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.475 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 --2- 172.21.15.90:0/4056265761 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800fe970 0x7f6f800fed70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 -- 172.21.15.90:0/4056265761 >> 172.21.15.90:0/4056265761 conn(0x7f6f800fa0f0 msgr2=0x7f6f800fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 -- 172.21.15.90:0/4056265761 shutdown_connections 2024-09-13T15:33:05.476 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.474+0000 7f6f84b11640 1 -- 172.21.15.90:0/4056265761 wait complete. 2024-09-13T15:33:05.477 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 Processor -- start 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 -- start start 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f800fe970 0x7f6f80195400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 0x7f6f80195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f80195e80 con 0x7f6f800ffb60 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f80195ff0 con 0x7f6f800fe970 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 0x7f6f80195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 0x7f6f80195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55682/0 (socket says 172.21.15.90:55682) 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 -- 172.21.15.90:0/390461454 learned_addr learned my addr 172.21.15.90:0/390461454 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7e59c640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f800fe970 0x7f6f80195400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 -- 172.21.15.90:0/390461454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f800fe970 msgr2=0x7f6f80195400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f800fe970 0x7f6f80195400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 -- 172.21.15.90:0/390461454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f70009560 con 0x7f6f800ffb60 2024-09-13T15:33:05.478 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f7dd9b640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 0x7f6f80195940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800d6f0 tx=0x7f6f6800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f68009a30 con 0x7f6f800ffb60 2024-09-13T15:33:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8019aac0 con 0x7f6f800ffb60 2024-09-13T15:33:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6f680054b0 con 0x7f6f800ffb60 2024-09-13T15:33:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f68004d00 con 0x7f6f800ffb60 2024-09-13T15:33:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.475+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8019b070 con 0x7f6f800ffb60 2024-09-13T15:33:05.479 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.476+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f800fed70 con 0x7f6f800ffb60 2024-09-13T15:33:05.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.477+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6f68005020 con 0x7f6f800ffb60 2024-09-13T15:33:05.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.477+0000 7f6f777fe640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6f6c06cc50 0x7f6f6c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.477+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f6f6808c440 con 0x7f6f800ffb60 2024-09-13T15:33:05.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.477+0000 7f6f7e59c640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6f6c06cc50 0x7f6f6c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.480 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.478+0000 7f6f7e59c640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6f6c06cc50 0x7f6f6c06f110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f70006010 tx=0x7f6f70004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:05.482 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.482+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6f680561a0 con 0x7f6f800ffb60 2024-09-13T15:33:05.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.658+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7f6f8019b4d0 con 0x7f6f800ffb60 2024-09-13T15:33:05.660 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.659+0000 7f6f777fe640 1 -- 172.21.15.90:0/390461454 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6f680597c0 con 0x7f6f800ffb60 2024-09-13T15:33:05.660 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6f6c06cc50 msgr2=0x7f6f6c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6f6c06cc50 0x7f6f6c06f110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6f70006010 tx=0x7f6f70004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 msgr2=0x7f6f80195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 0x7f6f80195940 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800d6f0 tx=0x7f6f6800dbb0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 shutdown_connections 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6f6c06cc50 0x7f6f6c06f110 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6f800ffb60 0x7f6f80195940 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 --2- 172.21.15.90:0/390461454 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f800fe970 0x7f6f80195400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 >> 172.21.15.90:0/390461454 conn(0x7f6f800fa0f0 msgr2=0x7f6f80067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 shutdown_connections 2024-09-13T15:33:05.663 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.661+0000 7f6f84b11640 1 -- 172.21.15.90:0/390461454 wait complete. 2024-09-13T15:33:05.673 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:33:05.786 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 -- 172.21.15.90:0/3380758090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 msgr2=0x7f90a01027e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3380758090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a01027e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f909c009880 tx=0x7f909c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 -- 172.21.15.90:0/3380758090 shutdown_connections 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3380758090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90a0102d20 0x7f90a0105110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3380758090 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a01027e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 -- 172.21.15.90:0/3380758090 >> 172.21.15.90:0/3380758090 conn(0x7f90a00f9f50 msgr2=0x7f90a00fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 -- 172.21.15.90:0/3380758090 shutdown_connections 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 -- 172.21.15.90:0/3380758090 wait complete. 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 Processor -- start 2024-09-13T15:33:05.787 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.784+0000 7f90a73a0640 1 -- start start 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a73a0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a0195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a73a0640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90a0102d20 0x7f90a0195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a73a0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0195d90 con 0x7f90a01003f0 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a73a0640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0195f00 con 0x7f90a0102d20 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a0195310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a0195310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55698/0 (socket says 172.21.15.90:55698) 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 -- 172.21.15.90:0/3945083820 learned_addr learned my addr 172.21.15.90:0/3945083820 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a5b9d640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90a0102d20 0x7f90a0195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 -- 172.21.15.90:0/3945083820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90a0102d20 msgr2=0x7f90a0195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90a0102d20 0x7f90a0195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.788 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 -- 172.21.15.90:0/3945083820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c009560 con 0x7f90a01003f0 2024-09-13T15:33:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a639e640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a0195310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f909c004f60 tx=0x7f909c004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909c0054e0 con 0x7f90a01003f0 2024-09-13T15:33:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a019a920 con 0x7f90a01003f0 2024-09-13T15:33:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f909c005640 con 0x7f90a01003f0 2024-09-13T15:33:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f909c03c960 con 0x7f90a01003f0 2024-09-13T15:33:05.789 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.785+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a019ae70 con 0x7f90a01003f0 2024-09-13T15:33:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.786+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a01007d0 con 0x7f90a01003f0 2024-09-13T15:33:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.787+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f909c003cf0 con 0x7f90a01003f0 2024-09-13T15:33:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.787+0000 7f90977fe640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f908c06cc50 0x7f908c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.787+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f909c0adba0 con 0x7f90a01003f0 2024-09-13T15:33:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.787+0000 7f90a5b9d640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f908c06cc50 0x7f908c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:05.790 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.788+0000 7f90a5b9d640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f908c06cc50 0x7f908c06f110 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9090005fd0 tx=0x7f9090005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:05.796 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.795+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f909c077850 con 0x7f90a01003f0 2024-09-13T15:33:05.970 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.968+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7f90a019b2e0 con 0x7f90a01003f0 2024-09-13T15:33:05.971 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.969+0000 7f90977fe640 1 -- 172.21.15.90:0/3945083820 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f909c07ae70 con 0x7f90a01003f0 2024-09-13T15:33:05.971 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.971+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f908c06cc50 msgr2=0x7f908c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.971+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f908c06cc50 0x7f908c06f110 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9090005fd0 tx=0x7f9090005850 comp rx=0 tx=0).stop 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.971+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 msgr2=0x7f90a0195310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.971+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a0195310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f909c004f60 tx=0x7f909c004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.971+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 shutdown_connections 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.971+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f908c06cc50 0x7f908c06f110 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.973 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.972+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90a0102d20 0x7f90a0195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.972+0000 7f90a73a0640 1 --2- 172.21.15.90:0/3945083820 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f90a01003f0 0x7f90a0195310 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:05.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.972+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 >> 172.21.15.90:0/3945083820 conn(0x7f90a00f9f50 msgr2=0x7f90a00fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:05.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.972+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 shutdown_connections 2024-09-13T15:33:05.974 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:05.972+0000 7f90a73a0640 1 -- 172.21.15.90:0/3945083820 wait complete. 2024-09-13T15:33:05.984 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:05 smithi090 ceph-mon[27811]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s wr, 13 op/s 2024-09-13T15:33:05.984 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:05 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/461384083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:05.984 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:05 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2560699658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:05.984 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:05 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/390461454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:05.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 -- 172.21.15.90:0/77050617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 msgr2=0x7f28c00fe190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/77050617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c00fe190 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009880 tx=0x7f28b802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 -- 172.21.15.90:0/77050617 shutdown_connections 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/77050617 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28c0100e00 0x7f28c0101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/77050617 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c00fe190 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 -- 172.21.15.90:0/77050617 >> 172.21.15.90:0/77050617 conn(0x7f28c00f9870 msgr2=0x7f28c00fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 -- 172.21.15.90:0/77050617 shutdown_connections 2024-09-13T15:33:06.091 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 -- 172.21.15.90:0/77050617 wait complete. 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 Processor -- start 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.088+0000 7f28c6f5a640 1 -- start start 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c6f5a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c0195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c6f5a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28c0100e00 0x7f28c0195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c6f5a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0195ec0 con 0x7f28c00fdd90 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c6f5a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c0196030 con 0x7f28c0100e00 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c0195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c0195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55718/0 (socket says 172.21.15.90:55718) 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 -- 172.21.15.90:0/3185068742 learned_addr learned my addr 172.21.15.90:0/3185068742 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28bffff640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28c0100e00 0x7f28c0195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 -- 172.21.15.90:0/3185068742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28c0100e00 msgr2=0x7f28c0195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.092 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28c0100e00 0x7f28c0195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 -- 172.21.15.90:0/3185068742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b8009560 con 0x7f28c00fdd90 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c4cf6640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c0195440 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009230 tx=0x7f28b8005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28b8005630 con 0x7f28c00fdd90 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f28b8003a80 con 0x7f28c00fdd90 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c019aa50 con 0x7f28c00fdd90 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.089+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f28b803cb10 con 0x7f28c00fdd90 2024-09-13T15:33:06.093 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.090+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c019afa0 con 0x7f28c00fdd90 2024-09-13T15:33:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.091+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c00fe190 con 0x7f28c00fdd90 2024-09-13T15:33:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.091+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f28b80035f0 con 0x7f28c00fdd90 2024-09-13T15:33:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.091+0000 7f28bdffb640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f28ac06cc60 0x7f28ac06f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.092+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f28b80adc70 con 0x7f28c00fdd90 2024-09-13T15:33:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.092+0000 7f28bffff640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f28ac06cc60 0x7f28ac06f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.094 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.092+0000 7f28bffff640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f28ac06cc60 0x7f28ac06f120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f28c0196900 tx=0x7f28b000a570 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.096+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f28b80779d0 con 0x7f28c00fdd90 2024-09-13T15:33:06.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:05 smithi175 ceph-mon[36128]: pgmap v230: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s wr, 13 op/s 2024-09-13T15:33:06.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:05 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/461384083' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:06.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:05 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2560699658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:06.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:05 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/390461454' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.298+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28c01084e0 con 0x7f28c00fdd90 2024-09-13T15:33:06.299 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.298+0000 7f28bdffb640 1 -- 172.21.15.90:0/3185068742 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4462 (secure 0 0 0) 0x7f28b807aff0 con 0x7f28c00fdd90 2024-09-13T15:33:06.300 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:06.300 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T15:32:52.637540+0000","modified":"2024-09-13T15:33:00.767625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14514},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14514":{"gid":14514,"name":"foo.smithi090.zyijng","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.90:6835/2333604411","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6834","nonce":2333604411},{"type":"v1","addr":"172.21.15.90:6835","nonce":2333604411}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"foo","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1},{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T15:32:58.742805+0000","modified":"2024-09-13T15:33:00.767625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24281":{"gid":24281,"name":"foo.smithi175.cvvnzp","rank":0,"incarnation":8,"state":"up:active","state_seq":2,"addr":"172.21.15.175:6833/2042693132","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":2042693132},{"type":"v1","addr":"172.21.15.175:6833","nonce":2042693132}]},"join_fscid":1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"bar","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T15:33:06.300 INFO:teuthology.orchestra.run.smithi090.stderr:dumped fsmap epoch 9 2024-09-13T15:33:06.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f28ac06cc60 msgr2=0x7f28ac06f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f28ac06cc60 0x7f28ac06f120 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f28c0196900 tx=0x7f28b000a570 comp rx=0 tx=0).stop 2024-09-13T15:33:06.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 msgr2=0x7f28c0195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c0195440 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009230 tx=0x7f28b8005230 comp rx=0 tx=0).stop 2024-09-13T15:33:06.302 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 shutdown_connections 2024-09-13T15:33:06.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f28ac06cc60 0x7f28ac06f120 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28c0100e00 0x7f28c0195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 --2- 172.21.15.90:0/3185068742 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f28c00fdd90 0x7f28c0195440 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 >> 172.21.15.90:0/3185068742 conn(0x7f28c00f9870 msgr2=0x7f28c0104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 shutdown_connections 2024-09-13T15:33:06.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.300+0000 7f28c6f5a640 1 -- 172.21.15.90:0/3185068742 wait complete. 2024-09-13T15:33:06.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:33:06.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 -- 172.21.15.90:0/3685460946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 msgr2=0x7f85b8105230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 --2- 172.21.15.90:0/3685460946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8105230 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f85ac009880 tx=0x7f85ac02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 -- 172.21.15.90:0/3685460946 shutdown_connections 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 --2- 172.21.15.90:0/3685460946 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8105230 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 --2- 172.21.15.90:0/3685460946 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85b8100510 0x7f85b8102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 -- 172.21.15.90:0/3685460946 >> 172.21.15.90:0/3685460946 conn(0x7f85b80fa0f0 msgr2=0x7f85b80fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 -- 172.21.15.90:0/3685460946 shutdown_connections 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.412+0000 7f85bcd83640 1 -- 172.21.15.90:0/3685460946 wait complete. 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85bcd83640 1 Processor -- start 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85bcd83640 1 -- start start 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85bcd83640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85b8100510 0x7f85b81953f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85bcd83640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85bcd83640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8195e70 con 0x7f85b8102e40 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85bcd83640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8195fe0 con 0x7f85b8100510 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85b5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.416 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85b5d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55746/0 (socket says 172.21.15.90:55746) 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.413+0000 7f85b5d9b640 1 -- 172.21.15.90:0/4259509631 learned_addr learned my addr 172.21.15.90:0/4259509631 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85b659c640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85b8100510 0x7f85b81953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85b5d9b640 1 -- 172.21.15.90:0/4259509631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85b8100510 msgr2=0x7f85b81953f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85b5d9b640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85b8100510 0x7f85b81953f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85b5d9b640 1 -- 172.21.15.90:0/4259509631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac009560 con 0x7f85b8102e40 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85b5d9b640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8195930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f85ac006010 tx=0x7f85ac005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ac0054e0 con 0x7f85b8102e40 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b819aa50 con 0x7f85b8102e40 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85ac005640 con 0x7f85b8102e40 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85ac03c990 con 0x7f85b8102e40 2024-09-13T15:33:06.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.414+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b819afa0 con 0x7f85b8102e40 2024-09-13T15:33:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.415+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b81008f0 con 0x7f85b8102e40 2024-09-13T15:33:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.415+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f85ac0035f0 con 0x7f85b8102e40 2024-09-13T15:33:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.416+0000 7f85a77fe640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f85a006cc50 0x7f85a006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.416+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f85ac0adbd0 con 0x7f85b8102e40 2024-09-13T15:33:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.416+0000 7f85b659c640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f85a006cc50 0x7f85a006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.416+0000 7f85b659c640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f85a006cc50 0x7f85a006f110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f85a8005fd0 tx=0x7f85a8005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.420+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f85ac077880 con 0x7f85b8102e40 2024-09-13T15:33:06.602 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.601+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85b81084c0 con 0x7f85b8102e40 2024-09-13T15:33:06.603 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.602+0000 7f85a77fe640 1 -- 172.21.15.90:0/4259509631 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+21061 (secure 0 0 0) 0x7f85ac07aea0 con 0x7f85b8102e40 2024-09-13T15:33:06.604 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:06.604 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":69,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:00.747847+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f85a006cc50 msgr2=0x7f85a006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f85a006cc50 0x7f85a006f110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f85a8005fd0 tx=0x7f85a8005850 comp rx=0 tx=0).stop 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 msgr2=0x7f85b8195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8195930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f85ac006010 tx=0x7f85ac005e50 comp rx=0 tx=0).stop 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 shutdown_connections 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f85a006cc50 0x7f85a006f110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f85b8102e40 0x7f85b8195930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 --2- 172.21.15.90:0/4259509631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85b8100510 0x7f85b81953f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 >> 172.21.15.90:0/4259509631 conn(0x7f85b80fa0f0 msgr2=0x7f85b80fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.606 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 shutdown_connections 2024-09-13T15:33:06.607 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.604+0000 7f85bcd83640 1 -- 172.21.15.90:0/4259509631 wait complete. 2024-09-13T15:33:06.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T15:32:58.742805+0000', 'modified': '2024-09-13T15:33:00.767625+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24281': {'gid': 24281, 'name': 'foo.smithi175.cvvnzp', 'rank': 0, 'incarnation': 8, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.175:6833/2042693132', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.175:6832', 'nonce': 2042693132}, {'type': 'v1', 'addr': '172.21.15.175:6833', 'nonce': 2042693132}]}, 'join_fscid': 1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'bar', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T15:33:06.617 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T15:33:06.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.foo.smithi175.cvvnzp status 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 -- 172.21.15.90:0/2804223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 msgr2=0x7f11d8100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 --2- 172.21.15.90:0/2804223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d8100000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009880 tx=0x7f11c002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 -- 172.21.15.90:0/2804223 shutdown_connections 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 --2- 172.21.15.90:0/2804223 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d8100000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 --2- 172.21.15.90:0/2804223 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11d80fe990 0x7f11d80fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 -- 172.21.15.90:0/2804223 >> 172.21.15.90:0/2804223 conn(0x7f11d80fa0f0 msgr2=0x7f11d80fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 -- 172.21.15.90:0/2804223 shutdown_connections 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 -- 172.21.15.90:0/2804223 wait complete. 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 Processor -- start 2024-09-13T15:33:06.722 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 -- start start 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.719+0000 7f11ddf33640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11d80fe990 0x7f11d8067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ddf33640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ddf33640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d819a670 con 0x7f11d80ffb80 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ddf33640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d819a7b0 con 0x7f11d80fe990 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55760/0 (socket says 172.21.15.90:55760) 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 -- 172.21.15.90:0/574169484 learned_addr learned my addr 172.21.15.90:0/574169484 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11d77fe640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11d80fe990 0x7f11d8067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 -- 172.21.15.90:0/574169484 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11d80fe990 msgr2=0x7f11d8067fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11d80fe990 0x7f11d8067fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 -- 172.21.15.90:0/574169484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c0009560 con 0x7f11d80ffb80 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ceffd640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d819a130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f11c0004f60 tx=0x7f11c0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11d57fa640 1 -- 172.21.15.90:0/574169484 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c0005560 con 0x7f11d80ffb80 2024-09-13T15:33:06.723 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11d57fa640 1 -- 172.21.15.90:0/574169484 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f11c0003a80 con 0x7f11d80ffb80 2024-09-13T15:33:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d819aa00 con 0x7f11d80ffb80 2024-09-13T15:33:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11d57fa640 1 -- 172.21.15.90:0/574169484 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c003ca40 con 0x7f11d80ffb80 2024-09-13T15:33:06.724 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.720+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d819af00 con 0x7f11d80ffb80 2024-09-13T15:33:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.722+0000 7f11d57fa640 1 -- 172.21.15.90:0/574169484 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f11c0003bf0 con 0x7f11d80ffb80 2024-09-13T15:33:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.722+0000 7f11d57fa640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f11c406cc50 0x7f11c406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.722+0000 7f11d57fa640 1 -- 172.21.15.90:0/574169484 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f11c00adae0 con 0x7f11d80ffb80 2024-09-13T15:33:06.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.722+0000 7f11d77fe640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f11c406cc50 0x7f11c406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.722+0000 7f11d77fe640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f11c406cc50 0x7f11c406f110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11c8005fd0 tx=0x7f11c800b410 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11e0c5b740 1 Processor -- start 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11e0c5b740 1 -- start start 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928911b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928961b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581927fac60 con 0x558192890db0 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581927f83f0 con 0x5581928916f0 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928911b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928911b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55774/0 (socket says 172.21.15.90:55774) 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 -- 172.21.15.90:0/1365216091 learned_addr learned my addr 172.21.15.90:0/1365216091 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11ceffd640 1 --2- 172.21.15.90:0/1365216091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 -- 172.21.15.90:0/1365216091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 msgr2=0x5581928961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 --2- 172.21.15.90:0/1365216091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 -- 172.21.15.90:0/1365216091 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581927b5f70 con 0x558192890db0 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.723+0000 7f11d77fe640 1 --2- 172.21.15.90:0/1365216091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928911b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f11c8003ce0 tx=0x7f11c8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8b40639835e1f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11cdffb640 1 -- 172.21.15.90:0/1365216091 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c8003ee0 con 0x558192890db0 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11cdffb640 1 -- 172.21.15.90:0/1365216091 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f11c8015070 con 0x558192890db0 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11cdffb640 1 -- 172.21.15.90:0/1365216091 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c8026610 con 0x558192890db0 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1365216091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 msgr2=0x5581928911b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1365216091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928911b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f11c8003ce0 tx=0x7f11c8013000 comp rx=0 tx=0).stop 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1365216091 shutdown_connections 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1365216091 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1365216091 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928911b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.731 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1365216091 >> 172.21.15.90:0/1365216091 conn(0x55819288c460 msgr2=0x55819288c860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1365216091 shutdown_connections 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.724+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1365216091 wait complete. 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 1 -- start start 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 1 Processor -- start 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x55819289a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x55819289a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581927f3ce0 con 0x5581928916f0 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581927b2300 con 0x558192890db0 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11e0c5b740 10 client.0 _ll_register_callbacks cb 0x55819277de00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x55819289a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x55819289a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55780/0 (socket says 172.21.15.90:55780) 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 -- 172.21.15.90:0/3805914122 learned_addr learned my addr 172.21.15.90:0/3805914122 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11d77fe640 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x55819289a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 -- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 msgr2=0x55819289a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x55819289a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 -- 172.21.15.90:0/3805914122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c8003990 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11ceffd640 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x55819289a6c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005ec0 tx=0x7f11c007afa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72383745c8c0a58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.725+0000 7f11b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11b6ffd640 1 -- 172.21.15.90:0/3805914122 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c0077e60 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11b6ffd640 1 -- 172.21.15.90:0/3805914122 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f11c0083430 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55819281c7c0 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11b6ffd640 1 -- 172.21.15.90:0/3805914122 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c00835f0 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 10 client.14634 fetch_fsmap learned FSMap version 9 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=3+}) v3 -- 0x5581927f2360 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11b6ffd640 1 -- 172.21.15.90:0/3805914122 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f11c0087080 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11b6ffd640 1 -- 172.21.15.90:0/3805914122 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) v1 ==== 2620+0+0 (secure 0 0 0) 0x7f11c0083ba0 con 0x5581928916f0 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 10 client.14634 fetch_fsmap finished waiting for FSMap version 9 2024-09-13T15:33:06.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 10 client.14634 resolve_mds: resolved name 'foo.smithi175.cvvnzp' to daemon mds.foo.smithi175.cvvnzp 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 20 client.14634 populate_metadata read hostname 'smithi090' 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928a3970 0x5581928a5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 4 client.14634 mds_command: new command op to 24281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 --> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581927e7da0 con 0x5581928a3970 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.726+0000 7f11d77fe640 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928a3970 0x5581928a5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.727+0000 7f11b6ffd640 10 client.14634 ms_handle_connect on v2:172.21.15.175:6832/2042693132 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.727+0000 7f11d77fe640 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928a3970 0x5581928a5d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be99f593612e47d6 server_cookie=84be6a49cf70a973 in_seq=0 out_seq=0 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.727+0000 7f11b6ffd640 1 -- 172.21.15.90:0/3805914122 <== mds.0 v2:172.21.15.175:6832/2042693132 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5581927e7da0 con 0x5581928a3970 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.727+0000 7f11b6ffd640 10 client.14634 handle_command_reply: tid=0 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.727+0000 7f11e0c5b740 1 client.14634 shutdown 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 msgr2=0x55819289a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.734 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x55819289a6c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f11c0005ec0 tx=0x7f11c007afa0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 shutdown_connections 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928a3970 0x5581928a5d60 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x55819289a6c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/3805914122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x55819289a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 >> 172.21.15.90:0/3805914122 conn(0x55819288c460 msgr2=0x55819288f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11b6ffd640 0 client.14634 ms_handle_reset on v2:172.21.15.175:6832/2042693132 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 shutdown_connections 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 1 -- 172.21.15.90:0/3805914122 wait complete. 2024-09-13T15:33:06.735 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.728+0000 7f11e0c5b740 20 client.14634 trim_cache size 0 max 16384 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11e0c5b740 1 Processor -- start 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11e0c5b740 1 -- start start 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x5581928b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x5581928b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581927b1fa0 con 0x5581928916f0 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581927775b0 con 0x558192890db0 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x5581928b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11ceffd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x5581928b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55790/0 (socket says 172.21.15.90:55790) 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.742+0000 7f11ceffd640 1 -- 172.21.15.90:0/286948985 learned_addr learned my addr 172.21.15.90:0/286948985 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.748 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11d77fe640 1 --2- 172.21.15.90:0/286948985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x5581928b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ceffd640 1 -- 172.21.15.90:0/286948985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 msgr2=0x5581928b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ceffd640 1 --2- 172.21.15.90:0/286948985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x5581928b3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ceffd640 1 -- 172.21.15.90:0/286948985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581927946e0 con 0x5581928916f0 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ceffd640 1 --2- 172.21.15.90:0/286948985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x5581928b40e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f11c0083f20 tx=0x7f11c0079b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa87810dc23ca0a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ce7fc640 1 -- 172.21.15.90:0/286948985 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c00812e0 con 0x5581928916f0 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ce7fc640 1 -- 172.21.15.90:0/286948985 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55819281c7c0 con 0x5581928916f0 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11ce7fc640 1 -- 172.21.15.90:0/286948985 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c0083660 con 0x5581928916f0 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 -- 172.21.15.90:0/286948985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 msgr2=0x5581928b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/286948985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x5581928b40e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f11c0083f20 tx=0x7f11c0079b30 comp rx=0 tx=0).stop 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 -- 172.21.15.90:0/286948985 shutdown_connections 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/286948985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x5581928916f0 0x5581928b40e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.749 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/286948985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x558192890db0 0x5581928b3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 -- 172.21.15.90:0/286948985 >> 172.21.15.90:0/286948985 conn(0x55819288c460 msgr2=0x558192898680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 -- 172.21.15.90:0/286948985 shutdown_connections 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.743+0000 7f11e0c5b740 1 -- 172.21.15.90:0/286948985 wait complete. 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 1 -- start start 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 1 Processor -- start 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928f6850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928f8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581928b43e0 con 0x558192890db0 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581928f92e0 con 0x5581928916f0 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11e0c5b740 10 client.0 _ll_register_callbacks cb 0x7f11c80042c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11d77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928f6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11d77fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928f6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55800/0 (socket says 172.21.15.90:55800) 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11d77fe640 1 -- 172.21.15.90:0/1761092419 learned_addr learned my addr 172.21.15.90:0/1761092419 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:06.750 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11ceffd640 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928f8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11d77fe640 1 -- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 msgr2=0x5581928f8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11d77fe640 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928f8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11d77fe640 1 -- 172.21.15.90:0/1761092419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581927f2360 con 0x558192890db0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.744+0000 7f11ceffd640 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928f8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11d77fe640 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928f6850 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11c8003c80 tx=0x7f11c8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4dff21621c5115 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.90:3300/0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11b6ffd640 1 -- 172.21.15.90:0/1761092419 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11c802a040 con 0x558192890db0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11b6ffd640 1 -- 172.21.15.90:0/1761092419 <== mon.0 v2:172.21.15.90:3300/0 2 ==== osd_map(69..69 src has 1..69) v4 ==== 7138+0+0 (secure 0 0 0) 0x7f11c8017070 con 0x558192890db0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11c8015070 con 0x558192890db0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11b6ffd640 1 -- 172.21.15.90:0/1761092419 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11c8005210 con 0x558192890db0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 10 client.14638 fetch_fsmap learned FSMap version 9 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=3+}) v3 -- 0x5581927b9df0 con 0x558192890db0 2024-09-13T15:33:06.751 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11b6ffd640 1 -- 172.21.15.90:0/1761092419 <== mon.0 v2:172.21.15.90:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f11c8015070 con 0x558192890db0 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11b6ffd640 1 -- 172.21.15.90:0/1761092419 <== mon.0 v2:172.21.15.90:3300/0 5 ==== fsmap(e 9) v1 ==== 2620+0+0 (secure 0 0 0) 0x7f11c8031040 con 0x558192890db0 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 10 client.14638 fetch_fsmap finished waiting for FSMap version 9 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 10 client.14638 resolve_mds: resolved name 'foo.smithi175.cvvnzp' to daemon mds.foo.smithi175.cvvnzp 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 20 client.14638 populate_metadata read hostname 'smithi090' 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928fb080 0x5581928fd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 4 client.14638 mds_command: new command op to 24281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 --> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5581924e75a0 con 0x5581928fb080 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.745+0000 7f11ceffd640 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928fb080 0x5581928fd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11b6ffd640 10 client.14638 ms_handle_connect on v2:172.21.15.175:6832/2042693132 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11ceffd640 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928fb080 0x5581928fd4d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3398c70c180565b1 server_cookie=762c2278be67a370 in_seq=0 out_seq=0 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11b6ffd640 1 -- 172.21.15.90:0/1761092419 <== mds.0 v2:172.21.15.175:6832/2042693132 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x5581924e75a0 con 0x5581928fb080 2024-09-13T15:33:06.752 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11b6ffd640 10 client.14638 handle_command_reply: tid=0 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 client.14638 shutdown 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 msgr2=0x5581928f6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928f6850 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f11c8003c80 tx=0x7f11c8011e90 comp rx=0 tx=0).stop 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 shutdown_connections 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5581928916f0 0x5581928f8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.175:6832/2042693132,v1:172.21.15.175:6833/2042693132] conn(0x5581928fb080 0x5581928fd4d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 --2- 172.21.15.90:0/1761092419 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x558192890db0 0x5581928f6850 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11b6ffd640 0 client.14638 ms_handle_reset on v2:172.21.15.175:6832/2042693132 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 >> 172.21.15.90:0/1761092419 conn(0x55819288c460 msgr2=0x55819288b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 shutdown_connections 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 1 -- 172.21.15.90:0/1761092419 wait complete. 2024-09-13T15:33:06.753 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.746+0000 7f11e0c5b740 20 client.14638 trim_cache size 0 max 16384 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_fsid": "41124e1c-71e4-11ef-bceb-c7b262605968", 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "whoami": 0, 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 24281, 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "want_state": "up:active", 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "state": "up:active", 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "bar", 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "rank_uptime": 6.9787240109999997, 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "mdsmap_epoch": 9, 2024-09-13T15:33:06.754 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch": 69, 2024-09-13T15:33:06.755 INFO:teuthology.orchestra.run.smithi090.stdout: "osdmap_epoch_barrier": 69, 2024-09-13T15:33:06.755 INFO:teuthology.orchestra.run.smithi090.stdout: "uptime": 7.4233993040000001 2024-09-13T15:33:06.755 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-09-13T15:33:06.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f11c406cc50 msgr2=0x7f11c406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.755 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f11c406cc50 0x7f11c406f110 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f11c8005fd0 tx=0x7f11c800b410 comp rx=0 tx=0).stop 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 msgr2=0x7f11d819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d819a130 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f11c0004f60 tx=0x7f11c0004600 comp rx=0 tx=0).stop 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 shutdown_connections 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f11c406cc50 0x7f11c406f110 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f11d80ffb80 0x7f11d819a130 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 --2- 172.21.15.90:0/574169484 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11d80fe990 0x7f11d8067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 >> 172.21.15.90:0/574169484 conn(0x7f11d80fa0f0 msgr2=0x7f11d8102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 shutdown_connections 2024-09-13T15:33:06.756 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:06.747+0000 7f11ddf33640 1 -- 172.21.15.90:0/574169484 wait complete. 2024-09-13T15:33:06.760 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-13T15:33:06.789 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-13T15:33:06.789 INFO:tasks.ceph_fuse:config is {'client.0': {'cephfs_name': 'foo'}} 2024-09-13T15:33:06.789 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'foo'} 2024-09-13T15:33:06.789 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T15:33:06.789 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:06.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:06.789 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-09-13T15:33:06.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:06.806 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-09-13T15:33:06.872 INFO:teuthology.orchestra.run.smithi090.stderr:Cannot find device "ceph-brx" 2024-09-13T15:33:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:33:06.874 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-13T15:33:06.874 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-13T15:33:06.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-13T15:33:07.114 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:33:07.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3945083820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:07.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3185068742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-13T15:33:07.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:07.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:07.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:07.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi175.hcljbb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:33:07.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi175.hcljbb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:33:07.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:07.159 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:06 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/4259509631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:07.231 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3945083820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:07.231 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3185068742' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-09-13T15:33:07.231 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:07.231 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:07.231 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:07.232 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi175.hcljbb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-13T15:33:07.232 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.bar.smithi175.hcljbb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-13T15:33:07.232 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:07.232 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:06 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/4259509631' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Deploying daemon mds.bar.smithi175.hcljbb on smithi175 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s wr, 10 op/s 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Cluster is now healthy 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:boot 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Dropping low affinity active daemon mds.foo.smithi175.cvvnzp in favor of higher affinity standby. 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Replacing daemon mds.foo.smithi175.cvvnzp as rank 0 with standby daemon mds.bar.smithi090.snmuxm 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-13T15:33:08.198 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1 {bar:0=foo.smithi175.cvvnzp=up:active,foo:0=foo.smithi090.zyijng=up:active} 1 up:standby 2024-09-13T15:33:08.199 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "bar.smithi090.snmuxm"}]: dispatch 2024-09-13T15:33:08.199 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T15:33:08.199 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:07 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi090.snmuxm=up:replay,foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Deploying daemon mds.bar.smithi175.hcljbb on smithi175 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: pgmap v231: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s wr, 10 op/s 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Cluster is now healthy 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:boot 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Dropping low affinity active daemon mds.foo.smithi175.cvvnzp in favor of higher affinity standby. 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Replacing daemon mds.foo.smithi175.cvvnzp as rank 0 with standby daemon mds.bar.smithi090.snmuxm 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1 {bar:0=foo.smithi175.cvvnzp=up:active,foo:0=foo.smithi090.zyijng=up:active} 1 up:standby 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "bar.smithi090.snmuxm"}]: dispatch 2024-09-13T15:33:08.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: osdmap e70: 8 total, 8 up, 8 in 2024-09-13T15:33:08.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:07 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi090.snmuxm=up:replay,foo:0=foo.smithi090.zyijng=up:active} 2024-09-13T15:33:08.829 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.826+0000 7f82d5052640 1 -- 172.21.15.90:0/310941768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 msgr2=0x7f82d00fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.826+0000 7f82d5052640 1 --2- 172.21.15.90:0/310941768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d00fee50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009880 tx=0x7f82b802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 -- 172.21.15.90:0/310941768 shutdown_connections 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 --2- 172.21.15.90:0/310941768 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d00fee50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 --2- 172.21.15.90:0/310941768 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82d01049f0 0x7f82d0104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 -- 172.21.15.90:0/310941768 >> 172.21.15.90:0/310941768 conn(0x7f82d00758a0 msgr2=0x7f82d00fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 -- 172.21.15.90:0/310941768 shutdown_connections 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 -- 172.21.15.90:0/310941768 wait complete. 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.827+0000 7f82d5052640 1 Processor -- start 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82d5052640 1 -- start start 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82d5052640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d0196840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82d5052640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82d01049f0 0x7f82d0196d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82d5052640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0197310 con 0x7f82d00fe9f0 2024-09-13T15:33:08.830 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82d5052640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0197450 con 0x7f82d01049f0 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d0196840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d0196840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55814/0 (socket says 172.21.15.90:55814) 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 -- 172.21.15.90:0/2753766889 learned_addr learned my addr 172.21.15.90:0/2753766889 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ce59c640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82d01049f0 0x7f82d0196d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 -- 172.21.15.90:0/2753766889 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82d01049f0 msgr2=0x7f82d0196d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82d01049f0 0x7f82d0196d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 -- 172.21.15.90:0/2753766889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8009560 con 0x7f82d00fe9f0 2024-09-13T15:33:08.831 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.828+0000 7f82ced9d640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d0196840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d8d0 tx=0x7f82c800dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:08.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.829+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c8009c00 con 0x7f82d00fe9f0 2024-09-13T15:33:08.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.829+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f82c80044b0 con 0x7f82d00fe9f0 2024-09-13T15:33:08.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.829+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f82c80047a0 con 0x7f82d00fe9f0 2024-09-13T15:33:08.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.829+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d019f8b0 con 0x7f82d00fe9f0 2024-09-13T15:33:08.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.829+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d019fcc0 con 0x7f82d00fe9f0 2024-09-13T15:33:08.832 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.830+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0100130 con 0x7f82d00fe9f0 2024-09-13T15:33:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.830+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f82c8004020 con 0x7f82d00fe9f0 2024-09-13T15:33:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.831+0000 7f82c7fff640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f82bc06cc50 0x7f82bc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.831+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f82c808c4f0 con 0x7f82d00fe9f0 2024-09-13T15:33:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.831+0000 7f82ce59c640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f82bc06cc50 0x7f82bc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:08.833 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.831+0000 7f82ce59c640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f82bc06cc50 0x7f82bc06f110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009230 tx=0x7f82b80058d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:08.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:08.835+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f82c80561d0 con 0x7f82d00fe9f0 2024-09-13T15:33:09.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.044+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f82d00682e0 con 0x7f82d00fe9f0 2024-09-13T15:33:09.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.045+0000 7f82c7fff640 1 -- 172.21.15.90:0/2753766889 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+148 (secure 0 0 0) 0x7f82c80597f0 con 0x7f82d00fe9f0 2024-09-13T15:33:09.046 INFO:teuthology.orchestra.run.smithi090.stdout:name: foo, metadata pool: cephfs.foo.meta, data pools: [cephfs.foo.data ] 2024-09-13T15:33:09.046 INFO:teuthology.orchestra.run.smithi090.stdout:name: bar, metadata pool: cephfs.bar.meta, data pools: [cephfs.bar.data ] 2024-09-13T15:33:09.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f82bc06cc50 msgr2=0x7f82bc06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:09.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f82bc06cc50 0x7f82bc06f110 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f82b8009230 tx=0x7f82b80058d0 comp rx=0 tx=0).stop 2024-09-13T15:33:09.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 msgr2=0x7f82d0196840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:09.048 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d0196840 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f82c800d8d0 tx=0x7f82c800dd90 comp rx=0 tx=0).stop 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 shutdown_connections 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f82bc06cc50 0x7f82bc06f110 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f82d01049f0 0x7f82d0196d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 --2- 172.21.15.90:0/2753766889 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f82d00fe9f0 0x7f82d0196840 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 >> 172.21.15.90:0/2753766889 conn(0x7f82d00758a0 msgr2=0x7f82d0102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 shutdown_connections 2024-09-13T15:33:09.049 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:09.047+0000 7f82d5052640 1 -- 172.21.15.90:0/2753766889 wait complete. 2024-09-13T15:33:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:08 smithi090 ceph-mon[27811]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-13T15:33:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:08 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:reconnect 2024-09-13T15:33:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:08 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.175:6832/2949616131,v1:172.21.15.175:6833/2949616131] up:boot 2024-09-13T15:33:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:08 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi090.snmuxm=up:reconnect,foo:0=foo.smithi090.zyijng=up:active} 1 up:standby 2024-09-13T15:33:09.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:08 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "foo.smithi175.cvvnzp"}]: dispatch 2024-09-13T15:33:09.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:08 smithi175 ceph-mon[36128]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-13T15:33:09.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:08 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:reconnect 2024-09-13T15:33:09.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:08 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.175:6832/2949616131,v1:172.21.15.175:6833/2949616131] up:boot 2024-09-13T15:33:09.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:08 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi090.snmuxm=up:reconnect,foo:0=foo.smithi090.zyijng=up:active} 1 up:standby 2024-09-13T15:33:09.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:08 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "foo.smithi175.cvvnzp"}]: dispatch 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:Mounting Ceph FS foo; just confirmed its presence on cluster 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi090.front.sepia.ceph.com 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:self.cephfs_name = foo 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T15:33:09.675 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T15:33:09.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:09.676 DEBUG:teuthology.orchestra.run.smithi090:> ip addr 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 ::1/128 scope host 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-09-13T15:33:09.692 INFO:teuthology.orchestra.run.smithi090.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a8 brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f0 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:d8:a1:a9 brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: altname enp7s0f1 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T15:33:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:66 brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: link/ether 0c:c4:7a:8f:d7:67 brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: inet 172.21.15.90/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft 42314sec preferred_lft 42314sec 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: inet6 fe80::ec4:7aff:fe8f:d767/64 scope link noprefixroute 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: valid_lft forever preferred_lft forever 2024-09-13T15:33:09.694 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T15:33:09.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:09.694 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-09-13T15:33:09.695 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-13T15:33:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add name ceph-brx type bridge 2024-09-13T15:33:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr flush dev ceph-brx 2024-09-13T15:33:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set ceph-brx up 2024-09-13T15:33:09.695 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T15:33:09.695 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-13T15:33:09.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:09.902 DEBUG:teuthology.orchestra.run.smithi090:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T15:33:09.929 INFO:teuthology.orchestra.run.smithi090.stdout:1 2024-09-13T15:33:09.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:09.931 DEBUG:teuthology.orchestra.run.smithi090:> route 2024-09-13T15:33:10.013 INFO:teuthology.orchestra.run.smithi090.stdout:Kernel IP routing table 2024-09-13T15:33:10.013 INFO:teuthology.orchestra.run.smithi090.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T15:33:10.013 INFO:teuthology.orchestra.run.smithi090.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T15:33:10.013 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-13T15:33:10.013 INFO:teuthology.orchestra.run.smithi090.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T15:33:10.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:10.014 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-09-13T15:33:10.014 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-13T15:33:10.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T15:33:10.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T15:33:10.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T15:33:10.014 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-13T15:33:10.075 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-13T15:33:10.075 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:rejoin 2024-09-13T15:33:10.075 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.175:6834/3402910592,v1:172.21.15.175:6835/3402910592] up:boot 2024-09-13T15:33:10.075 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi090.snmuxm=up:rejoin,foo:0=foo.smithi090.zyijng=up:active} 2 up:standby 2024-09-13T15:33:10.075 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "bar.smithi175.hcljbb"}]: dispatch 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: daemon mds.bar.smithi090.snmuxm is now active in filesystem bar as rank 0 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2753766889' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-13T15:33:10.076 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:09 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:33:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: pgmap v233: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-13T15:33:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:rejoin 2024-09-13T15:33:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.175:6834/3402910592,v1:172.21.15.175:6835/3402910592] up:boot 2024-09-13T15:33:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1/1 {bar:0=bar.smithi090.snmuxm=up:rejoin,foo:0=foo.smithi090.zyijng=up:active} 2 up:standby 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "mds metadata", "who": "bar.smithi175.hcljbb"}]: dispatch 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: daemon mds.bar.smithi090.snmuxm is now active in filesystem bar as rank 0 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2753766889' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-13T15:33:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:09 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:33:10.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:10.421 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-09-13T15:33:10.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:10.477 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list-id 2024-09-13T15:33:10.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:10.532 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-09-13T15:33:10.532 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-13T15:33:10.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T15:33:10.532 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-13T15:33:10.532 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-13T15:33:10.609 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-13T15:33:10.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:10.609 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-09-13T15:33:10.609 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-13T15:33:10.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-13T15:33:10.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T15:33:10.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-13T15:33:10.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-13T15:33:10.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-13T15:33:10.610 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-13T15:33:10.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:10.755 DEBUG:teuthology.orchestra.run.smithi090:> (cd / && exec bash -c ' 2024-09-13T15:33:10.755 DEBUG:teuthology.orchestra.run.smithi090:> set -e 2024-09-13T15:33:10.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set brx.0 up 2024-09-13T15:33:10.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T15:33:10.755 DEBUG:teuthology.orchestra.run.smithi090:> ') 2024-09-13T15:33:10.793 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'foo'} 2024-09-13T15:33:10.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T15:33:10.793 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:10.848 INFO:teuthology.orchestra.run.smithi090.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T15:33:10.849 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T15:33:10.849 DEBUG:teuthology.orchestra.run.smithi090:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:10.905 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-09-13T15:33:10.970 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T15:33:10.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T15:33:11.035 INFO:teuthology.orchestra.run.smithi090.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T15:33:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T15:33:11.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:11.037 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-09-13T15:33:11.094 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T15:33:11.094 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=foo) 2024-09-13T15:33:11.136 DEBUG:teuthology.orchestra.run.smithi090:> sudo modprobe fuse 2024-09-13T15:33:11.160 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T15:33:11.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T15:33:11.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:10 smithi090 ceph-mon[27811]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-13T15:33:11.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:10 smithi090 ceph-mon[27811]: Cluster is now healthy 2024-09-13T15:33:11.174 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:10 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:active 2024-09-13T15:33:11.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:10 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1 {bar:0=bar.smithi090.snmuxm=up:active,foo:0=foo.smithi090.zyijng=up:active} 2 up:standby 2024-09-13T15:33:11.198 INFO:teuthology.orchestra.run.smithi090.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T15:33:11.199 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T15:33:11.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:11.199 DEBUG:teuthology.orchestra.run.smithi090:> ls /sys/fs/fuse/connections 2024-09-13T15:33:11.247 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:2024-09-13T15:33:11.246+0000 7f3692d35480 -1 init, newargv = 0x562b0fab2ce0 newargc=15 2024-09-13T15:33:11.248 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[78972]: starting ceph client 2024-09-13T15:33:11.252 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[78972]: starting fuse 2024-09-13T15:33:11.256 INFO:teuthology.orchestra.run.smithi090.stdout:168 2024-09-13T15:33:11.257 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [168] 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> import glob 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> import re 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> import os 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> import subprocess 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> def _find_admin_socket(client_name): 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> files = glob.glob(asok_path) 2024-09-13T15:33:11.257 DEBUG:teuthology.orchestra.run.smithi090:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> # Given a non-glob path, it better be there 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> if "*" not in asok_path: 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> assert(len(files) == 1) 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> return files[0] 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> for f in files: 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T15:33:11.258 DEBUG:teuthology.orchestra.run.smithi090:> contents = proc_f.read() 2024-09-13T15:33:11.259 DEBUG:teuthology.orchestra.run.smithi090:> if mountpoint in contents: 2024-09-13T15:33:11.259 DEBUG:teuthology.orchestra.run.smithi090:> return f 2024-09-13T15:33:11.259 DEBUG:teuthology.orchestra.run.smithi090:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T15:33:11.259 DEBUG:teuthology.orchestra.run.smithi090:> 2024-09-13T15:33:11.259 DEBUG:teuthology.orchestra.run.smithi090:> print(_find_admin_socket("client.0")) 2024-09-13T15:33:11.259 DEBUG:teuthology.orchestra.run.smithi090:> ' 2024-09-13T15:33:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:10 smithi175 ceph-mon[36128]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-13T15:33:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:10 smithi175 ceph-mon[36128]: Cluster is now healthy 2024-09-13T15:33:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:10 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:active 2024-09-13T15:33:11.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:10 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1 {bar:0=bar.smithi090.snmuxm=up:active,foo:0=foo.smithi090.zyijng=up:active} 2 up:standby 2024-09-13T15:33:11.371 INFO:teuthology.orchestra.run.smithi090.stdout:/var/run/ceph/ceph-client.0.78972.asok 2024-09-13T15:33:11.558 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.78972.asok 2024-09-13T15:33:11.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:11.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.78972.asok status 2024-09-13T15:33:11.691 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2024-09-13T15:33:11.691 INFO:teuthology.orchestra.run.smithi090.stdout: "metadata": { 2024-09-13T15:33:11.691 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T15:33:11.691 INFO:teuthology.orchestra.run.smithi090.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "entity_id": "0", 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "hostname": "smithi090", 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "pid": "78972", 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "root": "/" 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_count": 0, 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "dentry_pinned_count": 0, 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "id": 14654, 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "inst": { 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "name": { 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "client", 2024-09-13T15:33:11.692 INFO:teuthology.orchestra.run.smithi090.stdout: "num": 14654 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2758808674 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": { 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "192.168.0.1:0", 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 2758808674 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2024-09-13T15:33:11.693 INFO:teuthology.orchestra.run.smithi090.stdout: "inst_str": "client.14654 192.168.0.1:0/2758808674", 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "addr_str": "192.168.0.1:0/2758808674", 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "inode_count": 1, 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_epoch": 9, 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch": 70, 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_epoch_barrier": 0, 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklisted": false, 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout: "fs_name": "foo" 2024-09-13T15:33:11.694 INFO:teuthology.orchestra.run.smithi090.stdout:} 2024-09-13T15:33:11.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:11.700 DEBUG:teuthology.orchestra.run.smithi090:> stat --file-system '--printf=%T 2024-09-13T15:33:11.700 DEBUG:teuthology.orchestra.run.smithi090:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:11.715 INFO:teuthology.orchestra.run.smithi090.stdout:fuseblk 2024-09-13T15:33:11.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:11.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:11.716 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:11.783 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-13T15:33:11.791 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-13T15:33:11.791 INFO:tasks.ceph_fuse:config is {'client.1': {'cephfs_name': 'bar'}} 2024-09-13T15:33:11.792 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'bar'} 2024-09-13T15:33:11.792 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T15:33:11.792 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:11.792 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:33:11.792 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:33:11.841 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:33:11.841 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:33:11.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:33:12.040 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.037+0000 7f70eaa19640 1 -- 172.21.15.90:0/3566610448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40ffb80 msgr2=0x7f70e4100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.037+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3566610448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40ffb80 0x7f70e4100000 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009880 tx=0x7f70d802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.037+0000 7f70eaa19640 1 -- 172.21.15.90:0/3566610448 shutdown_connections 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.037+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3566610448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40ffb80 0x7f70e4100000 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.037+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3566610448 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70e40fe990 0x7f70e40fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.037+0000 7f70eaa19640 1 -- 172.21.15.90:0/3566610448 >> 172.21.15.90:0/3566610448 conn(0x7f70e40fa0f0 msgr2=0x7f70e40fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 -- 172.21.15.90:0/3566610448 shutdown_connections 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 -- 172.21.15.90:0/3566610448 wait complete. 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 Processor -- start 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 -- start start 2024-09-13T15:33:12.041 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 0x7f70e4067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70e40ffb80 0x7f70e419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e419a690 con 0x7f70e40fe990 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.038+0000 7f70eaa19640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e419a7d0 con 0x7f70e40ffb80 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 0x7f70e4067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 0x7f70e4067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55836/0 (socket says 172.21.15.90:55836) 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 -- 172.21.15.90:0/3088931549 learned_addr learned my addr 172.21.15.90:0/3088931549 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e37fe640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70e40ffb80 0x7f70e419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 -- 172.21.15.90:0/3088931549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70e40ffb80 msgr2=0x7f70e419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70e40ffb80 0x7f70e419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 -- 172.21.15.90:0/3088931549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8009560 con 0x7f70e40fe990 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e3fff640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 0x7f70e4067fc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f70d400d8d0 tx=0x7f70d400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70d4009a30 con 0x7f70e40fe990 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f70d4009b90 con 0x7f70e40fe990 2024-09-13T15:33:12.042 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e419aa80 con 0x7f70e40fe990 2024-09-13T15:33:12.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f70d400f5f0 con 0x7f70e40fe990 2024-09-13T15:33:12.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.039+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e419b030 con 0x7f70e40fe990 2024-09-13T15:33:12.043 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.040+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e41073c0 con 0x7f70e40fe990 2024-09-13T15:33:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.041+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f70d4009d00 con 0x7f70e40fe990 2024-09-13T15:33:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.041+0000 7f70e17fa640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f70cc06cc50 0x7f70cc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.041+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f70d408c790 con 0x7f70e40fe990 2024-09-13T15:33:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.041+0000 7f70e37fe640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f70cc06cc50 0x7f70cc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.044 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.042+0000 7f70e37fe640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f70cc06cc50 0x7f70cc06f110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f70d8000c00 tx=0x7f70d8004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.046 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.046+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f70d40564a0 con 0x7f70e40fe990 2024-09-13T15:33:12.217 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.216+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70e41075f0 con 0x7f70e40fe990 2024-09-13T15:33:12.219 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.218+0000 7f70e17fa640 1 -- 172.21.15.90:0/3088931549 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+21193 (secure 0 0 0) 0x7f70d4059ac0 con 0x7f70e40fe990 2024-09-13T15:33:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:12.220 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":70,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:06.895687+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.175:6832/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.175:6833/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f70cc06cc50 msgr2=0x7f70cc06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f70cc06cc50 0x7f70cc06f110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f70d8000c00 tx=0x7f70d8004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 msgr2=0x7f70e4067fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 0x7f70e4067fc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f70d400d8d0 tx=0x7f70d400dd90 comp rx=0 tx=0).stop 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 shutdown_connections 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f70cc06cc50 0x7f70cc06f110 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70e40ffb80 0x7f70e419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 --2- 172.21.15.90:0/3088931549 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f70e40fe990 0x7f70e4067fc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 >> 172.21.15.90:0/3088931549 conn(0x7f70e40fa0f0 msgr2=0x7f70e4102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.222 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 shutdown_connections 2024-09-13T15:33:12.223 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.220+0000 7f70eaa19640 1 -- 172.21.15.90:0/3088931549 wait complete. 2024-09-13T15:33:12.232 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:11 smithi090 ceph-mon[27811]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-09-13T15:33:12.232 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:12.232 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:11 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:12.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:33:12.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:33:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:11 smithi175 ceph-mon[36128]: pgmap v234: 129 pgs: 129 active+clean; 581 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-09-13T15:33:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:11 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 -- 172.21.15.90:0/1513558926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80ffb90 msgr2=0x7f03f8100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 --2- 172.21.15.90:0/1513558926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80ffb90 0x7f03f8100010 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009880 tx=0x7f03ec02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 -- 172.21.15.90:0/1513558926 shutdown_connections 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 --2- 172.21.15.90:0/1513558926 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80ffb90 0x7f03f8100010 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 --2- 172.21.15.90:0/1513558926 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80fe9a0 0x7f03f80feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 -- 172.21.15.90:0/1513558926 >> 172.21.15.90:0/1513558926 conn(0x7f03f80fa100 msgr2=0x7f03f80fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 -- 172.21.15.90:0/1513558926 shutdown_connections 2024-09-13T15:33:12.341 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 -- 172.21.15.90:0/1513558926 wait complete. 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 Processor -- start 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.338+0000 7f03fc85a640 1 -- start start 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03fc85a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 0x7f03f8068b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03fc85a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80ffb90 0x7f03f819c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03fc85a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8195fb0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03fc85a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f819cb10 con 0x7f03f80ffb90 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 0x7f03f8068b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 0x7f03f8068b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55848/0 (socket says 172.21.15.90:55848) 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 -- 172.21.15.90:0/2546471361 learned_addr learned my addr 172.21.15.90:0/2546471361 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f5d9b640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80ffb90 0x7f03f819c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 -- 172.21.15.90:0/2546471361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80ffb90 msgr2=0x7f03f819c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80ffb90 0x7f03f819c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.342 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 -- 172.21.15.90:0/2546471361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ec009560 con 0x7f03f80fe9a0 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f5d9b640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80ffb90 0x7f03f819c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03f659c640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 0x7f03f8068b40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03e800c8f0 tx=0x7f03e800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e80041f0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f03e8007e30 con 0x7f03f80fe9a0 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f03e80047a0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f819cd10 con 0x7f03f80fe9a0 2024-09-13T15:33:12.343 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.339+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f819d290 con 0x7f03f80fe9a0 2024-09-13T15:33:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.340+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f81073d0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.341+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f03e80079a0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.341+0000 7f03e77fe640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f03e006cc50 0x7f03e006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.342+0000 7f03f5d9b640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f03e006cc50 0x7f03e006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.342+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f03e808c6e0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.344 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.342+0000 7f03f5d9b640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f03e006cc50 0x7f03e006f110 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f03ec000c00 tx=0x7f03ec005e40 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.347 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.346+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f03e80563c0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.509+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03f819d560 con 0x7f03f80fe9a0 2024-09-13T15:33:12.510 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.509+0000 7f03e77fe640 1 -- 172.21.15.90:0/2546471361 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f03e80599e0 con 0x7f03f80fe9a0 2024-09-13T15:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:33:12.512 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f03e006cc50 msgr2=0x7f03e006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f03e006cc50 0x7f03e006f110 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f03ec000c00 tx=0x7f03ec005e40 comp rx=0 tx=0).stop 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 msgr2=0x7f03f8068b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 0x7f03f8068b40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03e800c8f0 tx=0x7f03e800cdb0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 shutdown_connections 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f03e006cc50 0x7f03e006f110 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03f80ffb90 0x7f03f819c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.513 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 --2- 172.21.15.90:0/2546471361 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03f80fe9a0 0x7f03f8068b40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 >> 172.21.15.90:0/2546471361 conn(0x7f03f80fa100 msgr2=0x7f03f8102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 shutdown_connections 2024-09-13T15:33:12.514 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.511+0000 7f03fc85a640 1 -- 172.21.15.90:0/2546471361 wait complete. 2024-09-13T15:33:12.525 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:33:12.634 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 -- 172.21.15.90:0/2178496834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740ffb60 msgr2=0x7fd4740fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 --2- 172.21.15.90:0/2178496834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740ffb60 0x7fd4740fffe0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd468009880 tx=0x7fd46802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 -- 172.21.15.90:0/2178496834 shutdown_connections 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 --2- 172.21.15.90:0/2178496834 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740ffb60 0x7fd4740fffe0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 --2- 172.21.15.90:0/2178496834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740fe970 0x7fd4740fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 -- 172.21.15.90:0/2178496834 >> 172.21.15.90:0/2178496834 conn(0x7fd4740fa0f0 msgr2=0x7fd4740fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 -- 172.21.15.90:0/2178496834 shutdown_connections 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.631+0000 7fd47b880640 1 -- 172.21.15.90:0/2178496834 wait complete. 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47b880640 1 Processor -- start 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47b880640 1 -- start start 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47b880640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 0x7fd474067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47b880640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740ffb60 0x7fd47419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.635 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47b880640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47419a650 con 0x7fd4740fe970 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47b880640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47419a790 con 0x7fd4740ffb60 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47961c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 0x7fd474067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47961c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 0x7fd474067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55876/0 (socket says 172.21.15.90:55876) 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47961c640 1 -- 172.21.15.90:0/1923721284 learned_addr learned my addr 172.21.15.90:0/1923721284 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd478e1b640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740ffb60 0x7fd47419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.632+0000 7fd47961c640 1 -- 172.21.15.90:0/1923721284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740ffb60 msgr2=0x7fd47419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd47961c640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740ffb60 0x7fd47419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd47961c640 1 -- 172.21.15.90:0/1923721284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd470009490 con 0x7fd4740fe970 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd478e1b640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740ffb60 0x7fd47419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd47961c640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 0x7fd474067fc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd47000e940 tx=0x7fd47000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd47000cd50 con 0x7fd4740fe970 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4700108c0 con 0x7fd4740fe970 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd468009560 con 0x7fd4740fe970 2024-09-13T15:33:12.636 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd470018750 con 0x7fd4740fe970 2024-09-13T15:33:12.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.633+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd47419ae90 con 0x7fd4740fe970 2024-09-13T15:33:12.637 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.634+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474107380 con 0x7fd4740fe970 2024-09-13T15:33:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.635+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd470010430 con 0x7fd4740fe970 2024-09-13T15:33:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.635+0000 7fd4667fc640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd46006cc50 0x7fd46006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.636+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7fd470014070 con 0x7fd4740fe970 2024-09-13T15:33:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.636+0000 7fd478e1b640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd46006cc50 0x7fd46006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.638 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.636+0000 7fd478e1b640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd46006cc50 0x7fd46006f110 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd468006010 tx=0x7fd468000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.642 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.641+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd470056d60 con 0x7fd4740fe970 2024-09-13T15:33:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.819+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7fd47419b0e0 con 0x7fd4740fe970 2024-09-13T15:33:12.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.820+0000 7fd4667fc640 1 -- 172.21.15.90:0/1923721284 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd47419b0e0 con 0x7fd4740fe970 2024-09-13T15:33:12.821 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd46006cc50 msgr2=0x7fd46006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd46006cc50 0x7fd46006f110 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd468006010 tx=0x7fd468000bc0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 msgr2=0x7fd474067fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 0x7fd474067fc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd47000e940 tx=0x7fd47000ee00 comp rx=0 tx=0).stop 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 shutdown_connections 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd46006cc50 0x7fd46006f110 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd4740ffb60 0x7fd47419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 --2- 172.21.15.90:0/1923721284 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd4740fe970 0x7fd474067fc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 >> 172.21.15.90:0/1923721284 conn(0x7fd4740fa0f0 msgr2=0x7fd474102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 shutdown_connections 2024-09-13T15:33:12.824 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.822+0000 7fd47b880640 1 -- 172.21.15.90:0/1923721284 wait complete. 2024-09-13T15:33:12.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:33:12.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 -- 172.21.15.90:0/229921767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0fe970 msgr2=0x7f7b7c0fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/229921767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c0fed70 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009880 tx=0x7f7b7802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 -- 172.21.15.90:0/229921767 shutdown_connections 2024-09-13T15:33:12.939 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/229921767 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c0fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/229921767 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c0fed70 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 -- 172.21.15.90:0/229921767 >> 172.21.15.90:0/229921767 conn(0x7f7b7c0fa0f0 msgr2=0x7f7b7c0fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 -- 172.21.15.90:0/229921767 shutdown_connections 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 -- 172.21.15.90:0/229921767 wait complete. 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 Processor -- start 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.936+0000 7f7b83cbe640 1 -- start start 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b83cbe640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b83cbe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b83cbe640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c19a7a0 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b83cbe640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c19a8e0 con 0x7f7b7c0fe970 2024-09-13T15:33:12.940 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c19a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55888/0 (socket says 172.21.15.90:55888) 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 -- 172.21.15.90:0/2315161409 learned_addr learned my addr 172.21.15.90:0/2315161409 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81a5a640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 -- 172.21.15.90:0/2315161409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0fe970 msgr2=0x7f7b7c067fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c067fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 -- 172.21.15.90:0/2315161409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b78009560 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81a5a640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b81259640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c19a260 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b420 tx=0x7f7b7000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b70011800 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c19ab90 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7b70011df0 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b70010e80 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.941 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.937+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c19b170 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.938+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c107400 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.942 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.939+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f7b70011960 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.939+0000 7f7b6effd640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f7b6806cc50 0x7f7b6806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:12.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.940+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f7b7008c340 con 0x7f7b7c0ffb60 2024-09-13T15:33:12.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.940+0000 7f7b81a5a640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f7b6806cc50 0x7f7b6806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:12.943 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.940+0000 7f7b81a5a640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f7b6806cc50 0x7f7b6806f110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b78000c00 tx=0x7f7b78004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:12.945 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:12.944+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f7b70055fa0 con 0x7f7b7c0ffb60 2024-09-13T15:33:13.115 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.114+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7f7b7c19b3c0 con 0x7f7b7c0ffb60 2024-09-13T15:33:13.116 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.115+0000 7f7b6effd640 1 -- 172.21.15.90:0/2315161409 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7b7c19b3c0 con 0x7f7b7c0ffb60 2024-09-13T15:33:13.116 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:13.118 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f7b6806cc50 msgr2=0x7f7b6806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f7b6806cc50 0x7f7b6806f110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7b78000c00 tx=0x7f7b78004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 msgr2=0x7f7b7c19a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c19a260 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7b7000b420 tx=0x7f7b7000b8e0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 shutdown_connections 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f7b6806cc50 0x7f7b6806f110 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7b7c0ffb60 0x7f7b7c19a260 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.119 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 --2- 172.21.15.90:0/2315161409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b7c0fe970 0x7f7b7c067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 >> 172.21.15.90:0/2315161409 conn(0x7f7b7c0fa0f0 msgr2=0x7f7b7c102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 shutdown_connections 2024-09-13T15:33:13.120 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.117+0000 7f7b83cbe640 1 -- 172.21.15.90:0/2315161409 wait complete. 2024-09-13T15:33:13.127 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:12 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3088931549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:13.127 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:12 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2546471361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:13.127 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:12 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1923721284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:13.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:33:13.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 -- 172.21.15.90:0/936654375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 msgr2=0x7ff654105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 --2- 172.21.15.90:0/936654375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654105210 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009880 tx=0x7ff63c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 -- 172.21.15.90:0/936654375 shutdown_connections 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 --2- 172.21.15.90:0/936654375 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654105210 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 --2- 172.21.15.90:0/936654375 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6541004f0 0x7ff6541028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 -- 172.21.15.90:0/936654375 >> 172.21.15.90:0/936654375 conn(0x7ff6540fa0f0 msgr2=0x7ff6540fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 -- 172.21.15.90:0/936654375 shutdown_connections 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.229+0000 7ff658ac9640 1 -- 172.21.15.90:0/936654375 wait complete. 2024-09-13T15:33:13.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff658ac9640 1 Processor -- start 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff658ac9640 1 -- start start 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff658ac9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6541004f0 0x7ff6541953f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff658ac9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff658ac9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654195e70 con 0x7ff654102e20 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff658ac9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654195fe0 con 0x7ff6541004f0 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55898/0 (socket says 172.21.15.90:55898) 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 -- 172.21.15.90:0/635766479 learned_addr learned my addr 172.21.15.90:0/635766479 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff65259c640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6541004f0 0x7ff6541953f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 -- 172.21.15.90:0/635766479 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6541004f0 msgr2=0x7ff6541953f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.233 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6541004f0 0x7ff6541953f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 -- 172.21.15.90:0/635766479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c009560 con 0x7ff654102e20 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.230+0000 7ff649d9b640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654195930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff63c006010 tx=0x7ff63c005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.231+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63c0054e0 con 0x7ff654102e20 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.231+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff63c005640 con 0x7ff654102e20 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.231+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65419aa50 con 0x7ff654102e20 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.231+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63c03c990 con 0x7ff654102e20 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.231+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65419afa0 con 0x7ff654102e20 2024-09-13T15:33:13.234 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.232+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6541084a0 con 0x7ff654102e20 2024-09-13T15:33:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.232+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7ff63c0035f0 con 0x7ff654102e20 2024-09-13T15:33:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.233+0000 7ff64bfff640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ff64006cc50 0x7ff64006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.233+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7ff63c0adb70 con 0x7ff654102e20 2024-09-13T15:33:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.233+0000 7ff65259c640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ff64006cc50 0x7ff64006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.235 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.233+0000 7ff65259c640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ff64006cc50 0x7ff64006f110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff644005fd0 tx=0x7ff64400b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:13.238 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.237+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7ff63c077880 con 0x7ff654102e20 2024-09-13T15:33:13.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:12 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3088931549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:13.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:12 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2546471361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:13.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:12 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1923721284' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:13.414 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.413+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7ff65419b400 con 0x7ff654102e20 2024-09-13T15:33:13.415 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.413+0000 7ff64bfff640 1 -- 172.21.15.90:0/635766479 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff63c07aea0 con 0x7ff654102e20 2024-09-13T15:33:13.415 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:13.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ff64006cc50 msgr2=0x7ff64006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ff64006cc50 0x7ff64006f110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff644005fd0 tx=0x7ff64400b390 comp rx=0 tx=0).stop 2024-09-13T15:33:13.417 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 msgr2=0x7ff654195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654195930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff63c006010 tx=0x7ff63c005e50 comp rx=0 tx=0).stop 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 shutdown_connections 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7ff64006cc50 0x7ff64006f110 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7ff654102e20 0x7ff654195930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 --2- 172.21.15.90:0/635766479 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff6541004f0 0x7ff6541953f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 >> 172.21.15.90:0/635766479 conn(0x7ff6540fa0f0 msgr2=0x7ff6540fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 shutdown_connections 2024-09-13T15:33:13.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.416+0000 7ff658ac9640 1 -- 172.21.15.90:0/635766479 wait complete. 2024-09-13T15:33:13.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 -- 172.21.15.90:0/786280222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 msgr2=0x7fa78c0fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 --2- 172.21.15.90:0/786280222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c0fed90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa788009880 tx=0x7fa78802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 -- 172.21.15.90:0/786280222 shutdown_connections 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 --2- 172.21.15.90:0/786280222 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa78c0ffb80 0x7fa78c100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 --2- 172.21.15.90:0/786280222 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c0fed90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 -- 172.21.15.90:0/786280222 >> 172.21.15.90:0/786280222 conn(0x7fa78c0fa0f0 msgr2=0x7fa78c0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 -- 172.21.15.90:0/786280222 shutdown_connections 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 -- 172.21.15.90:0/786280222 wait complete. 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.530+0000 7fa7939e9640 1 Processor -- start 2024-09-13T15:33:13.533 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 -- start start 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa78c0ffb80 0x7fa78c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c19a740 con 0x7fa78c0fe990 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c19a880 con 0x7fa78c0ffb80 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c068b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55922/0 (socket says 172.21.15.90:55922) 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 -- 172.21.15.90:0/2605061503 learned_addr learned my addr 172.21.15.90:0/2605061503 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa790f84640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa78c0ffb80 0x7fa78c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 -- 172.21.15.90:0/2605061503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa78c0ffb80 msgr2=0x7fa78c19a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa78c0ffb80 0x7fa78c19a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 -- 172.21.15.90:0/2605061503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa788009560 con 0x7fa78c0fe990 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa791785640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c068b60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa788006010 tx=0x7fa788004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:13.534 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa788005460 con 0x7fa78c0fe990 2024-09-13T15:33:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa7880055c0 con 0x7fa78c0fe990 2024-09-13T15:33:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c19aad0 con 0x7fa78c0fe990 2024-09-13T15:33:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa78803c8e0 con 0x7fa78c0fe990 2024-09-13T15:33:13.535 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.531+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c19b050 con 0x7fa78c0fe990 2024-09-13T15:33:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.532+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78c1073c0 con 0x7fa78c0fe990 2024-09-13T15:33:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.533+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa7880035f0 con 0x7fa78c0fe990 2024-09-13T15:33:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.533+0000 7fa77e7fc640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa77806cc50 0x7fa77806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.533+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7fa7880adb00 con 0x7fa78c0fe990 2024-09-13T15:33:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.533+0000 7fa790f84640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa77806cc50 0x7fa77806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.536 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.534+0000 7fa790f84640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa77806cc50 0x7fa77806f110 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa780005fd0 tx=0x7fa780005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:13.538 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.537+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa7880777e0 con 0x7fa78c0fe990 2024-09-13T15:33:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.708+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7fa78c19b530 con 0x7fa78c0fe990 2024-09-13T15:33:13.709 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.708+0000 7fa77e7fc640 1 -- 172.21.15.90:0/2605061503 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa7880abe60 con 0x7fa78c0fe990 2024-09-13T15:33:13.710 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa77806cc50 msgr2=0x7fa77806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa77806cc50 0x7fa77806f110 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa780005fd0 tx=0x7fa780005850 comp rx=0 tx=0).stop 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 msgr2=0x7fa78c068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c068b60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa788006010 tx=0x7fa788004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 shutdown_connections 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa77806cc50 0x7fa77806f110 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa78c0ffb80 0x7fa78c19a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 --2- 172.21.15.90:0/2605061503 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa78c0fe990 0x7fa78c068b60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.712 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.710+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 >> 172.21.15.90:0/2605061503 conn(0x7fa78c0fa0f0 msgr2=0x7fa78c102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.711+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 shutdown_connections 2024-09-13T15:33:13.713 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.711+0000 7fa7939e9640 1 -- 172.21.15.90:0/2605061503 wait complete. 2024-09-13T15:33:13.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T15:33:13.835 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 -- 172.21.15.90:0/7323443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380fe9a0 msgr2=0x7fa1380feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/7323443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380fe9a0 0x7fa1380feda0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa134009880 tx=0x7fa13402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 -- 172.21.15.90:0/7323443 shutdown_connections 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/7323443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1380ffb90 0x7fa138100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/7323443 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380fe9a0 0x7fa1380feda0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 -- 172.21.15.90:0/7323443 >> 172.21.15.90:0/7323443 conn(0x7fa1380fa100 msgr2=0x7fa1380fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 -- 172.21.15.90:0/7323443 shutdown_connections 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.834+0000 7fa13f2f0640 1 -- 172.21.15.90:0/7323443 wait complete. 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13f2f0640 1 Processor -- start 2024-09-13T15:33:13.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13f2f0640 1 -- start start 2024-09-13T15:33:13.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13f2f0640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1380fe9a0 0x7fa138068ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13f2f0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 0x7fa13819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13f2f0640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13819a6a0 con 0x7fa1380ffb90 2024-09-13T15:33:13.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13f2f0640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13819a7e0 con 0x7fa1380fe9a0 2024-09-13T15:33:13.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 0x7fa13819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.838 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 0x7fa13819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:55932/0 (socket says 172.21.15.90:55932) 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 -- 172.21.15.90:0/3714213315 learned_addr learned my addr 172.21.15.90:0/3714213315 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 -- 172.21.15.90:0/3714213315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1380fe9a0 msgr2=0x7fa138068ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13d08c640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1380fe9a0 0x7fa138068ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1380fe9a0 0x7fa138068ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 -- 172.21.15.90:0/3714213315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa134009560 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.835+0000 7fa13c88b640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 0x7fa13819a160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c8f0 tx=0x7fa12800cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.836+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa128007b80 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.836+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa128007ce0 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.836+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa128004e10 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.836+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13819aa90 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.836+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13819b070 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.837+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1381073d0 con 0x7fa1380ffb90 2024-09-13T15:33:13.839 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.837+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fa128005020 con 0x7fa1380ffb90 2024-09-13T15:33:13.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.838+0000 7fa12e7fc640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa12406cc50 0x7fa12406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:13.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.838+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7fa12808c7c0 con 0x7fa1380ffb90 2024-09-13T15:33:13.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.838+0000 7fa13d08c640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa12406cc50 0x7fa12406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:13.841 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.838+0000 7fa13d08c640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa12406cc50 0x7fa12406f110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa134005ec0 tx=0x7fa134000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:13.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:13.844+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fa1280564a0 con 0x7fa1380ffb90 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: pgmap v235: 129 pgs: 129 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.175:6834/3402910592,v1:172.21.15.175:6835/3402910592] up:standby 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:active 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: fsmap foo:1 bar:1 {bar:0=bar.smithi090.snmuxm=up:active,foo:0=foo.smithi090.zyijng=up:active} 2 up:standby 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2315161409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/635766479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:14.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:13 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2605061503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:14.065 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.064+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa13819b2f0 con 0x7fa1380ffb90 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.065+0000 7fa12e7fc640 1 -- 172.21.15.90:0/3714213315 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v70) v1 ==== 81+0+1010 (secure 0 0 0) 0x7fa12801d410 con 0x7fa1380ffb90 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6832/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6833/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3168219877 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3860788333 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/575512074 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:14.066 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2223577157 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/782157526 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1586604027 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3782750394 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2606530289 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/4162862940 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:14.067 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:listed 17 entries 2024-09-13T15:33:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa12406cc50 msgr2=0x7fa12406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa12406cc50 0x7fa12406f110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa134005ec0 tx=0x7fa134000bc0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 msgr2=0x7fa13819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.068 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 0x7fa13819a160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa12800c8f0 tx=0x7fa12800cdb0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 shutdown_connections 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fa12406cc50 0x7fa12406f110 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fa1380ffb90 0x7fa13819a160 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 --2- 172.21.15.90:0/3714213315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa1380fe9a0 0x7fa138068ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.066+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 >> 172.21.15.90:0/3714213315 conn(0x7fa1380fa100 msgr2=0x7fa138102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.067+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 shutdown_connections 2024-09-13T15:33:14.069 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.067+0000 7fa13f2f0640 1 -- 172.21.15.90:0/3714213315 wait complete. 2024-09-13T15:33:14.081 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:33:14.081 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:33:14.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:33:14.239 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- 172.21.15.90:0/3944671980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4102e50 msgr2=0x7fdfa4105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/3944671980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4102e50 0x7fdfa4105240 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c009880 tx=0x7fdf8c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- 172.21.15.90:0/3944671980 shutdown_connections 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/3944671980 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4102e50 0x7fdfa4105240 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/3944671980 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfa4100520 0x7fdfa4102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- 172.21.15.90:0/3944671980 >> 172.21.15.90:0/3944671980 conn(0x7fdfa40fa100 msgr2=0x7fdfa40fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- 172.21.15.90:0/3944671980 shutdown_connections 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- 172.21.15.90:0/3944671980 wait complete. 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 Processor -- start 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- start start 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 0x7fdfa4195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfa4102e50 0x7fdfa4195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.240 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4195ed0 con 0x7fdfa4100520 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.237+0000 7fdfaaa53640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4196040 con 0x7fdfa4102e50 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 0x7fdfa4195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 0x7fdfa4195450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39574/0 (socket says 172.21.15.90:39574) 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 -- 172.21.15.90:0/2292524337 learned_addr learned my addr 172.21.15.90:0/2292524337 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdf9bfff640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfa4102e50 0x7fdfa4195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 -- 172.21.15.90:0/2292524337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfa4102e50 msgr2=0x7fdfa4195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfa4102e50 0x7fdfa4195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 -- 172.21.15.90:0/2292524337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c009560 con 0x7fdfa4100520 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa3fff640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 0x7fdfa4195450 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400c8f0 tx=0x7fdf9400cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf94007b80 con 0x7fdfa4100520 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdf94007ce0 con 0x7fdfa4100520 2024-09-13T15:33:14.241 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa419ab10 con 0x7fdfa4100520 2024-09-13T15:33:14.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf94004e10 con 0x7fdfa4100520 2024-09-13T15:33:14.242 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.238+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa419b040 con 0x7fdfa4100520 2024-09-13T15:33:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.239+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa4100900 con 0x7fdfa4100520 2024-09-13T15:33:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.240+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fdf94005020 con 0x7fdfa4100520 2024-09-13T15:33:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.240+0000 7fdfa1ffb640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fdf9006cc50 0x7fdf9006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.240+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7fdf9408c790 con 0x7fdfa4100520 2024-09-13T15:33:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.240+0000 7fdf9bfff640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fdf9006cc50 0x7fdf9006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.243 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.240+0000 7fdf9bfff640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fdf9006cc50 0x7fdf9006f110 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c000c00 tx=0x7fdf8c004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:14.245 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.244+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fdf940563f0 con 0x7fdfa4100520 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: pgmap v235: 129 pgs: 129 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.175:6834/3402910592,v1:172.21.15.175:6835/3402910592] up:standby 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: mds.? [v2:172.21.15.90:6836/2072095155,v1:172.21.15.90:6837/2072095155] up:active 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: fsmap foo:1 bar:1 {bar:0=bar.smithi090.snmuxm=up:active,foo:0=foo.smithi090.zyijng=up:active} 2 up:standby 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2315161409' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/635766479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:14.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:13 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2605061503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:14.407 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.406+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfa41084d0 con 0x7fdfa4100520 2024-09-13T15:33:14.409 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.408+0000 7fdfa1ffb640 1 -- 172.21.15.90:0/2292524337 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+21193 (secure 0 0 0) 0x7fdf94059a10 con 0x7fdfa4100520 2024-09-13T15:33:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:33:14.409 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":70,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:06.895687+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.175:6832/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.175:6833/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fdf9006cc50 msgr2=0x7fdf9006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fdf9006cc50 0x7fdf9006f110 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c000c00 tx=0x7fdf8c004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 msgr2=0x7fdfa4195450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 0x7fdfa4195450 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400c8f0 tx=0x7fdf9400cdb0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 shutdown_connections 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fdf9006cc50 0x7fdf9006f110 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdfa4102e50 0x7fdfa4195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.412 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 --2- 172.21.15.90:0/2292524337 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fdfa4100520 0x7fdfa4195450 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 >> 172.21.15.90:0/2292524337 conn(0x7fdfa40fa100 msgr2=0x7fdfa40fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 shutdown_connections 2024-09-13T15:33:14.413 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.410+0000 7fdfaaa53640 1 -- 172.21.15.90:0/2292524337 wait complete. 2024-09-13T15:33:14.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:33:14.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:33:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 -- 172.21.15.90:0/133419251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 msgr2=0x7f26940feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 --2- 172.21.15.90:0/133419251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26940feda0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2684009880 tx=0x7f268402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 -- 172.21.15.90:0/133419251 shutdown_connections 2024-09-13T15:33:14.526 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 --2- 172.21.15.90:0/133419251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26940ffb90 0x7f2694100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 --2- 172.21.15.90:0/133419251 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26940feda0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 -- 172.21.15.90:0/133419251 >> 172.21.15.90:0/133419251 conn(0x7f26940fa100 msgr2=0x7f26940fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 -- 172.21.15.90:0/133419251 shutdown_connections 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 -- 172.21.15.90:0/133419251 wait complete. 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.523+0000 7f269a4e9640 1 Processor -- start 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 -- start start 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26941954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26940ffb90 0x7f2694195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694195f40 con 0x7f26940fe9a0 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26941960b0 con 0x7f26940ffb90 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26941954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26941954c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39588/0 (socket says 172.21.15.90:39588) 2024-09-13T15:33:14.527 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 -- 172.21.15.90:0/2826100985 learned_addr learned my addr 172.21.15.90:0/2826100985 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f268bfff640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26940ffb90 0x7f2694195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 -- 172.21.15.90:0/2826100985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26940ffb90 msgr2=0x7f2694195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26940ffb90 0x7f2694195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 -- 172.21.15.90:0/2826100985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2684009560 con 0x7f26940fe9a0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2693fff640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26941954c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2684006010 tx=0x7f2684004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2684005460 con 0x7f26940fe9a0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f269419ab20 con 0x7f26940fe9a0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f26840055c0 con 0x7f26940fe9a0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f268403c8e0 con 0x7f26940fe9a0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.524+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269419b070 con 0x7f26940fe9a0 2024-09-13T15:33:14.528 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.525+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26941073d0 con 0x7f26940fe9a0 2024-09-13T15:33:14.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.526+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f26840035f0 con 0x7f26940fe9a0 2024-09-13T15:33:14.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.526+0000 7f2691ffb640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f268006cc50 0x7f268006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.526+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f26840adaf0 con 0x7f26940fe9a0 2024-09-13T15:33:14.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.526+0000 7f268bfff640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f268006cc50 0x7f268006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.529 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.527+0000 7f268bfff640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f268006cc50 0x7f268006f110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f26941969d0 tx=0x7f267c00b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:14.531 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.531+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f2684077800 con 0x7f26940fe9a0 2024-09-13T15:33:14.696 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.695+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f269419b5a0 con 0x7f26940fe9a0 2024-09-13T15:33:14.697 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.696+0000 7f2691ffb640 1 -- 172.21.15.90:0/2826100985 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f268407ae20 con 0x7f26940fe9a0 2024-09-13T15:33:14.697 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:33:14.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.697+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f268006cc50 msgr2=0x7f268006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.697+0000 7f269a4e9640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f268006cc50 0x7f268006f110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f26941969d0 tx=0x7f267c00b390 comp rx=0 tx=0).stop 2024-09-13T15:33:14.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 msgr2=0x7f26941954c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26941954c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2684006010 tx=0x7f2684004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 shutdown_connections 2024-09-13T15:33:14.699 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f268006cc50 0x7f268006f110 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f26940ffb90 0x7f2694195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 --2- 172.21.15.90:0/2826100985 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f26940fe9a0 0x7f26941954c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 >> 172.21.15.90:0/2826100985 conn(0x7f26940fa100 msgr2=0x7f2694067fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 shutdown_connections 2024-09-13T15:33:14.700 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.698+0000 7f269a4e9640 1 -- 172.21.15.90:0/2826100985 wait complete. 2024-09-13T15:33:14.711 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:33:14.817 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- 172.21.15.90:0/1893541232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00fe970 msgr2=0x7f8ec00fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/1893541232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00fe970 0x7f8ec00fed70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8009880 tx=0x7f8eb802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- 172.21.15.90:0/1893541232 shutdown_connections 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/1893541232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ec00ffb60 0x7f8ec00fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/1893541232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00fe970 0x7f8ec00fed70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- 172.21.15.90:0/1893541232 >> 172.21.15.90:0/1893541232 conn(0x7f8ec00fa0f0 msgr2=0x7f8ec00fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- 172.21.15.90:0/1893541232 shutdown_connections 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- 172.21.15.90:0/1893541232 wait complete. 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 Processor -- start 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- start start 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ec00fe970 0x7f8ec0067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 0x7f8ec019a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec019a690 con 0x7f8ec00ffb60 2024-09-13T15:33:14.818 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.815+0000 7f8ec86e9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec019a7d0 con 0x7f8ec00fe970 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 0x7f8ec019a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 0x7f8ec019a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39598/0 (socket says 172.21.15.90:39598) 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 -- 172.21.15.90:0/2542039790 learned_addr learned my addr 172.21.15.90:0/2542039790 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec6485640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ec00fe970 0x7f8ec0067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 -- 172.21.15.90:0/2542039790 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ec00fe970 msgr2=0x7f8ec0067fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ec00fe970 0x7f8ec0067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 -- 172.21.15.90:0/2542039790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb8009560 con 0x7f8ec00ffb60 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec5c84640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 0x7f8ec019a150 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d8d0 tx=0x7f8eb000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eb0009c00 con 0x7f8ec00ffb60 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec019aa80 con 0x7f8ec00ffb60 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8eb00044b0 con 0x7f8ec00ffb60 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8eb00047a0 con 0x7f8ec00ffb60 2024-09-13T15:33:14.819 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.816+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec019afe0 con 0x7f8ec00ffb60 2024-09-13T15:33:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.817+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec0107380 con 0x7f8ec00ffb60 2024-09-13T15:33:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.818+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f8eb0004020 con 0x7f8ec00ffb60 2024-09-13T15:33:14.820 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.818+0000 7f8eb77fe640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8eac06cc50 0x7f8eac06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.818+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f8eb008c700 con 0x7f8ec00ffb60 2024-09-13T15:33:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.818+0000 7f8ec6485640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8eac06cc50 0x7f8eac06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:14.821 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.819+0000 7f8ec6485640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8eac06cc50 0x7f8eac06f110 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005ec0 tx=0x7f8eb8000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:14.823 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.822+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f8eb0056360 con 0x7f8ec00ffb60 2024-09-13T15:33:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.986+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7f8ec019ac10 con 0x7f8ec00ffb60 2024-09-13T15:33:14.987 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.986+0000 7f8eb77fe640 1 -- 172.21.15.90:0/2542039790 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8ec019ac10 con 0x7f8ec00ffb60 2024-09-13T15:33:14.988 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:14.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8eac06cc50 msgr2=0x7f8eac06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8eac06cc50 0x7f8eac06f110 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8005ec0 tx=0x7f8eb8000bc0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 msgr2=0x7f8ec019a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 0x7f8ec019a150 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8eb000d8d0 tx=0x7f8eb000dd90 comp rx=0 tx=0).stop 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 shutdown_connections 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f8eac06cc50 0x7f8eac06f110 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f8ec00ffb60 0x7f8ec019a150 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 --2- 172.21.15.90:0/2542039790 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ec00fe970 0x7f8ec0067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 >> 172.21.15.90:0/2542039790 conn(0x7f8ec00fa0f0 msgr2=0x7f8ec0102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 shutdown_connections 2024-09-13T15:33:14.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:14.988+0000 7f8ec86e9640 1 -- 172.21.15.90:0/2542039790 wait complete. 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3714213315' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2292524337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:15.000 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: pgmap v236: 129 pgs: 129 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 102 B/s wr, 8 op/s 2024-09-13T15:33:15.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:14 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2826100985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:15.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:33:15.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.101+0000 7f79965ce640 1 -- 172.21.15.90:0/1987479029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 msgr2=0x7f79901051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.104 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.101+0000 7f79965ce640 1 --2- 172.21.15.90:0/1987479029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f79901051d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7978009880 tx=0x7f797802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.101+0000 7f79965ce640 1 -- 172.21.15.90:0/1987479029 shutdown_connections 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.101+0000 7f79965ce640 1 --2- 172.21.15.90:0/1987479029 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f79901051d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.101+0000 7f79965ce640 1 --2- 172.21.15.90:0/1987479029 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 0x7f79901028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 -- 172.21.15.90:0/1987479029 >> 172.21.15.90:0/1987479029 conn(0x7f79900fa090 msgr2=0x7f79900fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 -- 172.21.15.90:0/1987479029 shutdown_connections 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 -- 172.21.15.90:0/1987479029 wait complete. 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 Processor -- start 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 -- start start 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 0x7f7990195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f7990195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7990195ed0 con 0x7f7990102de0 2024-09-13T15:33:15.105 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f79965ce640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7990196040 con 0x7f79901004b0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f7987fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f7990195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f7987fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f7990195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39620/0 (socket says 172.21.15.90:39620) 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.102+0000 7f7987fff640 1 -- 172.21.15.90:0/1089554973 learned_addr learned my addr 172.21.15.90:0/1089554973 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f798ffff640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 0x7f7990195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f7987fff640 1 -- 172.21.15.90:0/1089554973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 msgr2=0x7f7990195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f7987fff640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 0x7f7990195450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f7987fff640 1 -- 172.21.15.90:0/1089554973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978009560 con 0x7f7990102de0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f798ffff640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 0x7f7990195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f7987fff640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f7990195990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7978006010 tx=0x7f7978004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7978005460 con 0x7f7990102de0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f79780055c0 con 0x7f7990102de0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799019aab0 con 0x7f7990102de0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f797803c8e0 con 0x7f7990102de0 2024-09-13T15:33:15.106 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.103+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799019b000 con 0x7f7990102de0 2024-09-13T15:33:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.104+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7990100890 con 0x7f7990102de0 2024-09-13T15:33:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.104+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f79780035f0 con 0x7f7990102de0 2024-09-13T15:33:15.107 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.105+0000 7f798dffb640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f797c06cc50 0x7f797c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.105+0000 7f798ffff640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f797c06cc50 0x7f797c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.105+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f79780adad0 con 0x7f7990102de0 2024-09-13T15:33:15.108 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.105+0000 7f798ffff640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f797c06cc50 0x7f797c06f110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7980009b40 tx=0x7f7980005e50 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.110 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.109+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f79780777b0 con 0x7f7990102de0 2024-09-13T15:33:15.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3714213315' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2292524337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: pgmap v236: 129 pgs: 129 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 102 B/s wr, 8 op/s 2024-09-13T15:33:15.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:14 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2826100985' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:33:15.272 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.271+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7f799019b4c0 con 0x7f7990102de0 2024-09-13T15:33:15.273 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.272+0000 7f798dffb640 1 -- 172.21.15.90:0/1089554973 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f797807add0 con 0x7f7990102de0 2024-09-13T15:33:15.273 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f797c06cc50 msgr2=0x7f797c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f797c06cc50 0x7f797c06f110 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7980009b40 tx=0x7f7980005e50 comp rx=0 tx=0).stop 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 msgr2=0x7f7990195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f7990195990 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7978006010 tx=0x7f7978004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 shutdown_connections 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f797c06cc50 0x7f797c06f110 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f7990102de0 0x7f7990195990 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 --2- 172.21.15.90:0/1089554973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f79901004b0 0x7f7990195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 >> 172.21.15.90:0/1089554973 conn(0x7f79900fa090 msgr2=0x7f79900fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 shutdown_connections 2024-09-13T15:33:15.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.274+0000 7f79965ce640 1 -- 172.21.15.90:0/1089554973 wait complete. 2024-09-13T15:33:15.286 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 -- 172.21.15.90:0/30246797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 msgr2=0x7fd290101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 --2- 172.21.15.90:0/30246797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd290101100 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd284009880 tx=0x7fd28402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 -- 172.21.15.90:0/30246797 shutdown_connections 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 --2- 172.21.15.90:0/30246797 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd290101100 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 --2- 172.21.15.90:0/30246797 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2900ffa90 0x7fd2900ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 -- 172.21.15.90:0/30246797 >> 172.21.15.90:0/30246797 conn(0x7fd2900fb1f0 msgr2=0x7fd2900fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 -- 172.21.15.90:0/30246797 shutdown_connections 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.385+0000 7fd295b5e640 1 -- 172.21.15.90:0/30246797 wait complete. 2024-09-13T15:33:15.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd295b5e640 1 Processor -- start 2024-09-13T15:33:15.389 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd295b5e640 1 -- start start 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd295b5e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2900ffa90 0x7fd290195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd295b5e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd2901959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd295b5e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290195ef0 con 0x7fd290100c80 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd295b5e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd290196060 con 0x7fd2900ffa90 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd28effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd2901959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd28effd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd2901959b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39630/0 (socket says 172.21.15.90:39630) 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd28effd640 1 -- 172.21.15.90:0/556028252 learned_addr learned my addr 172.21.15.90:0/556028252 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.386+0000 7fd28f7fe640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2900ffa90 0x7fd290195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28effd640 1 -- 172.21.15.90:0/556028252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2900ffa90 msgr2=0x7fd290195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28effd640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2900ffa90 0x7fd290195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28effd640 1 -- 172.21.15.90:0/556028252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284009560 con 0x7fd290100c80 2024-09-13T15:33:15.390 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28effd640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd2901959b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd284004f60 tx=0x7fd284004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2840054e0 con 0x7fd290100c80 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd284005640 con 0x7fd290100c80 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd28403c960 con 0x7fd290100c80 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29019aad0 con 0x7fd290100c80 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.387+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29019b020 con 0x7fd290100c80 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.388+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2901084c0 con 0x7fd290100c80 2024-09-13T15:33:15.391 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.389+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd284003cf0 con 0x7fd290100c80 2024-09-13T15:33:15.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.389+0000 7fd28cff9640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd27806cc50 0x7fd27806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.389+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7fd2840adb10 con 0x7fd290100c80 2024-09-13T15:33:15.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.389+0000 7fd28f7fe640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd27806cc50 0x7fd27806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.392 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.390+0000 7fd28f7fe640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd27806cc50 0x7fd27806f110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd280005fd0 tx=0x7fd28000b390 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.394 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.393+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd284077820 con 0x7fd290100c80 2024-09-13T15:33:15.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.554+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7fd29019b240 con 0x7fd290100c80 2024-09-13T15:33:15.556 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.555+0000 7fd28cff9640 1 -- 172.21.15.90:0/556028252 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd28407ae40 con 0x7fd290100c80 2024-09-13T15:33:15.556 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd27806cc50 msgr2=0x7fd27806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd27806cc50 0x7fd27806f110 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd280005fd0 tx=0x7fd28000b390 comp rx=0 tx=0).stop 2024-09-13T15:33:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 msgr2=0x7fd2901959b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd2901959b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd284004f60 tx=0x7fd284004da0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 shutdown_connections 2024-09-13T15:33:15.558 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd27806cc50 0x7fd27806f110 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd290100c80 0x7fd2901959b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 --2- 172.21.15.90:0/556028252 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd2900ffa90 0x7fd290195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 >> 172.21.15.90:0/556028252 conn(0x7fd2900fb1f0 msgr2=0x7fd290103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 shutdown_connections 2024-09-13T15:33:15.559 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.557+0000 7fd295b5e640 1 -- 172.21.15.90:0/556028252 wait complete. 2024-09-13T15:33:15.569 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:33:15.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.668+0000 7f35af072640 1 -- 172.21.15.90:0/2355761501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 msgr2=0x7f35a80fed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.668+0000 7f35af072640 1 --2- 172.21.15.90:0/2355761501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a80fed70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f35a4009880 tx=0x7f35a402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.668+0000 7f35af072640 1 -- 172.21.15.90:0/2355761501 shutdown_connections 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.668+0000 7f35af072640 1 --2- 172.21.15.90:0/2355761501 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35a80ffb60 0x7f35a80fffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.668+0000 7f35af072640 1 --2- 172.21.15.90:0/2355761501 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a80fed70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 -- 172.21.15.90:0/2355761501 >> 172.21.15.90:0/2355761501 conn(0x7f35a80fa0f0 msgr2=0x7f35a80fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 -- 172.21.15.90:0/2355761501 shutdown_connections 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 -- 172.21.15.90:0/2355761501 wait complete. 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 Processor -- start 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 -- start start 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a8067f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35a80ffb60 0x7f35a819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a819a650 con 0x7f35a80fe970 2024-09-13T15:33:15.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35af072640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a819a790 con 0x7f35a80ffb60 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35ace0e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a8067f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35ace0e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a8067f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39638/0 (socket says 172.21.15.90:39638) 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.669+0000 7f35ace0e640 1 -- 172.21.15.90:0/2959435320 learned_addr learned my addr 172.21.15.90:0/2959435320 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f35ace0e640 1 -- 172.21.15.90:0/2959435320 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35a80ffb60 msgr2=0x7f35a819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f35ace0e640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35a80ffb60 0x7f35a819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f35ace0e640 1 -- 172.21.15.90:0/2959435320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a4009560 con 0x7f35a80fe970 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f35ace0e640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a8067f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f35a4006010 tx=0x7f35a4005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a4005600 con 0x7f35a80fe970 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a819a9c0 con 0x7f35a80fe970 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f35a4003a80 con 0x7f35a80fe970 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f35a403ca90 con 0x7f35a80fe970 2024-09-13T15:33:15.673 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.670+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a819af40 con 0x7f35a80fe970 2024-09-13T15:33:15.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.671+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a81073a0 con 0x7f35a80fe970 2024-09-13T15:33:15.674 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.672+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f35a4003bf0 con 0x7f35a80fe970 2024-09-13T15:33:15.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.672+0000 7f359dffb640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f359006cc50 0x7f359006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.672+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7f35a40adcc0 con 0x7f35a80fe970 2024-09-13T15:33:15.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.672+0000 7f359ffff640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f359006cc50 0x7f359006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.675 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.672+0000 7f359ffff640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f359006cc50 0x7f359006f110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f35a8068530 tx=0x7f3598005bf0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.677 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.676+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f35a4077920 con 0x7f35a80fe970 2024-09-13T15:33:15.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.840+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7f35a819b440 con 0x7f35a80fe970 2024-09-13T15:33:15.842 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.841+0000 7f359dffb640 1 -- 172.21.15.90:0/2959435320 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f35a407af40 con 0x7f35a80fe970 2024-09-13T15:33:15.842 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:33:15.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f359006cc50 msgr2=0x7f359006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f359006cc50 0x7f359006f110 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f35a8068530 tx=0x7f3598005bf0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 msgr2=0x7f35a8067f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a8067f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f35a4006010 tx=0x7f35a4005000 comp rx=0 tx=0).stop 2024-09-13T15:33:15.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 shutdown_connections 2024-09-13T15:33:15.844 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f359006cc50 0x7f359006f110 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35a80ffb60 0x7f35a819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 --2- 172.21.15.90:0/2959435320 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f35a80fe970 0x7f35a8067f90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 >> 172.21.15.90:0/2959435320 conn(0x7f35a80fa0f0 msgr2=0x7f35a8102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 shutdown_connections 2024-09-13T15:33:15.845 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.843+0000 7f35af072640 1 -- 172.21.15.90:0/2959435320 wait complete. 2024-09-13T15:33:15.855 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 -- 172.21.15.90:0/2724786583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40fe9a0 msgr2=0x7fb9c40feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/2724786583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40fe9a0 0x7fb9c40feda0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009880 tx=0x7fb9b402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 -- 172.21.15.90:0/2724786583 shutdown_connections 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/2724786583 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/2724786583 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40fe9a0 0x7fb9c40feda0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 -- 172.21.15.90:0/2724786583 >> 172.21.15.90:0/2724786583 conn(0x7fb9c40fa100 msgr2=0x7fb9c40fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:15.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 -- 172.21.15.90:0/2724786583 shutdown_connections 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 -- 172.21.15.90:0/2724786583 wait complete. 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.954+0000 7fb9ca49c640 1 Processor -- start 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9ca49c640 1 -- start start 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9ca49c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c40fe9a0 0x7fb9c4102930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9ca49c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9ca49c640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4102e70 con 0x7fb9c40ffb90 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9ca49c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c41014e0 con 0x7fb9c40fe9a0 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39658/0 (socket says 172.21.15.90:39658) 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 -- 172.21.15.90:0/1489500416 learned_addr learned my addr 172.21.15.90:0/1489500416 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c3fff640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c40fe9a0 0x7fb9c4102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 -- 172.21.15.90:0/1489500416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c40fe9a0 msgr2=0x7fb9c4102930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:15.958 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c40fe9a0 0x7fb9c4102930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 -- 172.21.15.90:0/1489500416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b4009560 con 0x7fb9c40ffb90 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.955+0000 7fb9c37fe640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100f70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800b420 tx=0x7fb9b800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.956+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9b8011800 con 0x7fb9c40ffb90 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.956+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb9b8011df0 con 0x7fb9c40ffb90 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.956+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c41017c0 con 0x7fb9c40ffb90 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.956+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9b8010e80 con 0x7fb9c40ffb90 2024-09-13T15:33:15.959 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.956+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c4101da0 con 0x7fb9c40ffb90 2024-09-13T15:33:15.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.956+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c4107430 con 0x7fb9c40ffb90 2024-09-13T15:33:15.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.957+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb9b8011960 con 0x7fb9c40ffb90 2024-09-13T15:33:15.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.958+0000 7fb9c17fa640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb9ac06cc50 0x7fb9ac06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:15.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.958+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7fb9b808c340 con 0x7fb9c40ffb90 2024-09-13T15:33:15.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.958+0000 7fb9c3fff640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb9ac06cc50 0x7fb9ac06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:15.960 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.958+0000 7fb9c3fff640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb9ac06cc50 0x7fb9ac06f110 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4000c00 tx=0x7fb9b4004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:15.965 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:15.964+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb9b8055fa0 con 0x7fb9c40ffb90 2024-09-13T15:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.137+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb9c41075c0 con 0x7fb9c40ffb90 2024-09-13T15:33:16.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.137+0000 7fb9c17fa640 1 -- 172.21.15.90:0/1489500416 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v70) v1 ==== 81+0+1010 (secure 0 0 0) 0x7fb9b80595c0 con 0x7fb9c40ffb90 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6832/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6833/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3168219877 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3860788333 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/575512074 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2223577157 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:16.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/782157526 2024-09-14T15:25:26.727266+0000 2024-09-13T15:33:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1586604027 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3782750394 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2606530289 2024-09-14T15:26:07.051031+0000 2024-09-13T15:33:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/4162862940 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:16.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:listed 17 entries 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb9ac06cc50 msgr2=0x7fb9ac06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb9ac06cc50 0x7fb9ac06f110 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4000c00 tx=0x7fb9b4004f00 comp rx=0 tx=0).stop 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 msgr2=0x7fb9c4100f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100f70 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb9b800b420 tx=0x7fb9b800b8e0 comp rx=0 tx=0).stop 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 shutdown_connections 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb9ac06cc50 0x7fb9ac06f110 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:16.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb9c40ffb90 0x7fb9c4100f70 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 --2- 172.21.15.90:0/1489500416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9c40fe9a0 0x7fb9c4102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 >> 172.21.15.90:0/1489500416 conn(0x7fb9c40fa100 msgr2=0x7fb9c4067fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 shutdown_connections 2024-09-13T15:33:16.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:16.139+0000 7fb9ca49c640 1 -- 172.21.15.90:0/1489500416 wait complete. 2024-09-13T15:33:16.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2542039790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1089554973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/556028252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:15 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/2959435320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.151 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi090.front.sepia.ceph.com... 2024-09-13T15:33:16.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:16.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:16.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi090.stderr:ceph-fuse[78972]: fuse finished with error 0 and tester_r 0 2024-09-13T15:33:16.176 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T15:33:16.222 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi090.front.sepia.ceph.com 2024-09-13T15:33:16.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:16.222 DEBUG:teuthology.orchestra.run.smithi090:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T15:33:16.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:16.249 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-09-13T15:33:16.262 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:15 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2542039790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.262 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:15 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1089554973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.262 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:15 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/556028252' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.262 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:15 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/2959435320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:33:16.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:16.266 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link delete ceph-brx 2024-09-13T15:33:16.337 INFO:teuthology.orchestra.run.smithi175.stderr:Cannot find device "ceph-brx" 2024-09-13T15:33:16.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:33:16.338 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-13T15:33:16.338 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-13T15:33:16.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361 shell --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 -- ceph fs ls 2024-09-13T15:33:16.536 INFO:teuthology.orchestra.run.smithi090.stderr:Inferring config /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/mon.smithi090/config 2024-09-13T15:33:17.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:16 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1489500416' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:33:17.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:16 smithi175 ceph-mon[36128]: pgmap v237: 129 pgs: 129 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 102 B/s wr, 8 op/s 2024-09-13T15:33:17.237 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:16 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1489500416' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:33:17.237 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:16 smithi090 ceph-mon[27811]: pgmap v237: 129 pgs: 129 active+clean; 582 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 102 B/s wr, 8 op/s 2024-09-13T15:33:18.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 -- 172.21.15.90:0/3023430710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa41049f0 msgr2=0x7fafa4104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:18.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 --2- 172.21.15.90:0/3023430710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa41049f0 0x7fafa4104dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009880 tx=0x7faf9c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 -- 172.21.15.90:0/3023430710 shutdown_connections 2024-09-13T15:33:18.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 --2- 172.21.15.90:0/3023430710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafa40fe9f0 0x7fafa40fee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.438 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 --2- 172.21.15.90:0/3023430710 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa41049f0 0x7fafa4104dd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 -- 172.21.15.90:0/3023430710 >> 172.21.15.90:0/3023430710 conn(0x7fafa40758a0 msgr2=0x7fafa40fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:18.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 -- 172.21.15.90:0/3023430710 shutdown_connections 2024-09-13T15:33:18.439 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 -- 172.21.15.90:0/3023430710 wait complete. 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.437+0000 7fafa900f640 1 Processor -- start 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa900f640 1 -- start start 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa900f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 0x7fafa41967e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa900f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafa41049f0 0x7fafa4196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa900f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa4197260 con 0x7fafa40fe9f0 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa900f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa41973d0 con 0x7fafa41049f0 2024-09-13T15:33:18.440 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 0x7fafa41967e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 0x7fafa41967e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:39670/0 (socket says 172.21.15.90:39670) 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 -- 172.21.15.90:0/542663599 learned_addr learned my addr 172.21.15.90:0/542663599 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa259c640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafa41049f0 0x7fafa4196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 -- 172.21.15.90:0/542663599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafa41049f0 msgr2=0x7fafa4196d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafa41049f0 0x7fafa4196d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 -- 172.21.15.90:0/542663599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9c009560 con 0x7fafa40fe9f0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.438+0000 7fafa2d9d640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 0x7fafa41967e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faf9c006010 tx=0x7faf9c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.439+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf9c005460 con 0x7fafa40fe9f0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.439+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa4197650 con 0x7fafa40fe9f0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.439+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faf9c0055c0 con 0x7fafa40fe9f0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.439+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf9c03c900 con 0x7fafa40fe9f0 2024-09-13T15:33:18.441 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.439+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa419fb30 con 0x7fafa40fe9f0 2024-09-13T15:33:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.440+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa4100130 con 0x7fafa40fe9f0 2024-09-13T15:33:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.440+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7faf9c03cae0 con 0x7fafa40fe9f0 2024-09-13T15:33:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.440+0000 7faf9bfff640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faf9006cc50 0x7faf9006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:33:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.441+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7224+0+0 (secure 0 0 0) 0x7faf9c0ada60 con 0x7fafa40fe9f0 2024-09-13T15:33:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.441+0000 7fafa259c640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faf9006cc50 0x7faf9006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:33:18.442 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.441+0000 7fafa259c640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faf9006cc50 0x7faf9006f110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faf8c005fd0 tx=0x7faf8c005e20 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:33:18.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.445+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7faf9c077730 con 0x7fafa40fe9f0 2024-09-13T15:33:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.658+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fafa4197650 con 0x7fafa40fe9f0 2024-09-13T15:33:18.659 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.658+0000 7faf9bfff640 1 -- 172.21.15.90:0/542663599 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+148 (secure 0 0 0) 0x7fafa4197650 con 0x7fafa40fe9f0 2024-09-13T15:33:18.659 INFO:teuthology.orchestra.run.smithi090.stdout:name: foo, metadata pool: cephfs.foo.meta, data pools: [cephfs.foo.data ] 2024-09-13T15:33:18.660 INFO:teuthology.orchestra.run.smithi090.stdout:name: bar, metadata pool: cephfs.bar.meta, data pools: [cephfs.bar.data ] 2024-09-13T15:33:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faf9006cc50 msgr2=0x7faf9006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faf9006cc50 0x7faf9006f110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faf8c005fd0 tx=0x7faf8c005e20 comp rx=0 tx=0).stop 2024-09-13T15:33:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 msgr2=0x7fafa41967e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:33:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 0x7fafa41967e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faf9c006010 tx=0x7faf9c004610 comp rx=0 tx=0).stop 2024-09-13T15:33:18.661 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 shutdown_connections 2024-09-13T15:33:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7faf9006cc50 0x7faf9006f110 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafa41049f0 0x7fafa4196d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 --2- 172.21.15.90:0/542663599 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fafa40fe9f0 0x7fafa41967e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:33:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 >> 172.21.15.90:0/542663599 conn(0x7fafa40758a0 msgr2=0x7fafa40fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:33:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 shutdown_connections 2024-09-13T15:33:18.662 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:33:18.660+0000 7fafa900f640 1 -- 172.21.15.90:0/542663599 wait complete. 2024-09-13T15:33:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-09-13T15:33:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-13T15:33:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-13T15:33:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-13T15:33:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]: dispatch 2024-09-13T15:33:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-13T15:33:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-13T15:33:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-13T15:33:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:33:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:19.436 INFO:tasks.cephfs.mount:Mounting Ceph FS bar; just confirmed its presence on cluster 2024-09-13T15:33:19.436 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T15:33:19.436 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2024-09-13T15:33:19.436 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T15:33:19.436 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:19.437 INFO:tasks.cephfs.mount:self.cephfs_name = bar 2024-09-13T15:33:19.437 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T15:33:19.437 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T15:33:19.437 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T15:33:19.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:19.437 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2024-09-13T15:33:19.458 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T15:33:19.458 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T15:33:19.458 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T15:33:19.458 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-09-13T15:33:19.458 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-09-13T15:33:19.458 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-13T15:33:19.459 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42285sec preferred_lft 42285sec 2024-09-13T15:33:19.460 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link noprefixroute 2024-09-13T15:33:19.460 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-09-13T15:33:19.460 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T15:33:19.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:19.460 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2024-09-13T15:33:19.460 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-09-13T15:33:19.460 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add name ceph-brx type bridge 2024-09-13T15:33:19.460 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip addr flush dev ceph-brx 2024-09-13T15:33:19.460 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set ceph-brx up 2024-09-13T15:33:19.461 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T15:33:19.461 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-09-13T15:33:19.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:19.644 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T15:33:19.668 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-09-13T15:33:19.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:19.671 DEBUG:teuthology.orchestra.run.smithi175:> route 2024-09-13T15:33:19.739 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: pgmap v238: 129 pgs: 129 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 351 B/s wr, 7 op/s 2024-09-13T15:33:19.739 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-09-13T15:33:19.740 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-13T15:33:19.740 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T15:33:19.740 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T15:33:19.741 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T15:33:19.741 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/542663599' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-13T15:33:19.741 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:19 smithi090 ceph-mon[27811]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T15:33:19.743 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2024-09-13T15:33:19.744 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T15:33:19.744 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T15:33:19.744 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-13T15:33:19.744 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T15:33:19.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:19.745 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2024-09-13T15:33:19.745 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-09-13T15:33:19.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T15:33:19.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T15:33:19.745 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T15:33:19.746 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-09-13T15:33:19.806 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: pgmap v238: 129 pgs: 129 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 7.0 KiB/s rd, 351 B/s wr, 7 op/s 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [7, 2]}]': finished 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: osdmap e71: 8 total, 8 up, 8 in 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/542663599' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-09-13T15:33:19.807 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:19 smithi175 ceph-mon[36128]: osdmap e72: 8 total, 8 up, 8 in 2024-09-13T15:33:19.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:19.998 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-09-13T15:33:20.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:20.054 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list-id 2024-09-13T15:33:20.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:20.112 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2024-09-13T15:33:20.112 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-09-13T15:33:20.112 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T15:33:20.112 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-13T15:33:20.112 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-09-13T15:33:20.201 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-13T15:33:20.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-13T15:33:20.202 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-09-13T15:33:20.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:20.335 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec bash -c ' 2024-09-13T15:33:20.335 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-09-13T15:33:20.335 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.0 up 2024-09-13T15:33:20.336 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T15:33:20.336 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-09-13T15:33:20.373 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'bar'} 2024-09-13T15:33:20.373 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T15:33:20.374 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:20.431 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T15:33:20.431 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T15:33:20.431 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:20.486 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-09-13T15:33:20.553 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T15:33:20.553 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T15:33:20.618 INFO:teuthology.orchestra.run.smithi175.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T15:33:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T15:33:20.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:20.619 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-09-13T15:33:20.675 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T15:33:20.676 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=bar) 2024-09-13T15:33:20.718 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-09-13T15:33:20.742 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T15:33:20.742 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T15:33:20.807 INFO:teuthology.orchestra.run.smithi175.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T15:33:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T15:33:20.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:20.808 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-09-13T15:33:20.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:2024-09-13T15:33:20.820+0000 7f35a6ff9480 -1 init, newargv = 0x557c921fcc20 newargc=15 2024-09-13T15:33:20.822 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[65134]: starting ceph client 2024-09-13T15:33:20.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[65134]: starting fuse 2024-09-13T15:33:20.837 INFO:teuthology.orchestra.run.smithi175.stdout:132 2024-09-13T15:33:20.837 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [132] 2024-09-13T15:33:20.837 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> import glob 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> import re 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> import os 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> import subprocess 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> def _find_admin_socket(client_name): 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> files = glob.glob(asok_path) 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> # Given a non-glob path, it better be there 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> if "*" not in asok_path: 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> assert(len(files) == 1) 2024-09-13T15:33:20.838 DEBUG:teuthology.orchestra.run.smithi175:> return files[0] 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> for f in files: 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> contents = proc_f.read() 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> if mountpoint in contents: 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> return f 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> 2024-09-13T15:33:20.839 DEBUG:teuthology.orchestra.run.smithi175:> print(_find_admin_socket("client.1")) 2024-09-13T15:33:20.840 DEBUG:teuthology.orchestra.run.smithi175:> ' 2024-09-13T15:33:20.950 INFO:teuthology.orchestra.run.smithi175.stdout:/var/run/ceph/ceph-client.1.65134.asok 2024-09-13T15:33:21.138 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.65134.asok 2024-09-13T15:33:21.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:21.138 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.65134.asok status 2024-09-13T15:33:21.269 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "metadata": { 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "entity_id": "1", 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "hostname": "smithi175", 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "pid": "65134", 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: "root": "/" 2024-09-13T15:33:21.270 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_count": 0, 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_pinned_count": 0, 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 24421, 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "inst": { 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "name": { 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "client", 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "num": 24421 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-09-13T15:33:21.271 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 3382984526 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 3382984526 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "inst_str": "client.24421 192.168.0.1:0/3382984526", 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "addr_str": "192.168.0.1:0/3382984526", 2024-09-13T15:33:21.272 INFO:teuthology.orchestra.run.smithi175.stdout: "inode_count": 1, 2024-09-13T15:33:21.273 INFO:teuthology.orchestra.run.smithi175.stdout: "mds_epoch": 15, 2024-09-13T15:33:21.273 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch": 72, 2024-09-13T15:33:21.273 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch_barrier": 0, 2024-09-13T15:33:21.273 INFO:teuthology.orchestra.run.smithi175.stdout: "blocklisted": false, 2024-09-13T15:33:21.273 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "bar" 2024-09-13T15:33:21.273 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-09-13T15:33:21.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:21.278 DEBUG:teuthology.orchestra.run.smithi175:> stat --file-system '--printf=%T 2024-09-13T15:33:21.278 DEBUG:teuthology.orchestra.run.smithi175:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:21.294 INFO:teuthology.orchestra.run.smithi175.stdout:fuseblk 2024-09-13T15:33:21.294 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:21.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:33:21.294 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:21.360 INFO:teuthology.run_tasks:Running task workunit... 2024-09-13T15:33:21.375 INFO:tasks.workunit:Pulling workunits from ref 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:33:21.376 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-13T15:33:21.376 INFO:tasks.workunit:timeout=3h 2024-09-13T15:33:21.376 INFO:tasks.workunit:cleanup=True 2024-09-13T15:33:21.376 DEBUG:teuthology.orchestra.run.smithi090:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T15:33:21.395 INFO:teuthology.orchestra.run.smithi090.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T15:33:21.396 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:21.454 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-13T15:33:21.454 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-13T15:33:21.512 DEBUG:teuthology.orchestra.run.smithi175:> stat -- /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 84h/132d Inode: 1 Links: 2 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:fusefs_t:s0 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-09-13T15:33:21.529 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-09-13 15:32:59.769042628 +0000 2024-09-13T15:33:21.530 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-09-13 15:33:21.358146345 +0000 2024-09-13T15:33:21.530 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: - 2024-09-13T15:33:21.530 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-09-13T15:33:21.530 DEBUG:teuthology.orchestra.run.smithi175:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-09-13T15:33:21.598 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:33:21.599 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T15:33:21.618 INFO:tasks.workunit.client.0.smithi090.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-13T15:33:21.655 INFO:tasks.workunit.client.1.smithi175.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-09-13T15:33:21.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:21 smithi175 ceph-mon[36128]: pgmap v241: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 383 B/s wr, 7 op/s 2024-09-13T15:33:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:21 smithi090 ceph-mon[27811]: pgmap v241: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 223 MiB used, 715 GiB / 715 GiB avail; 7.6 KiB/s rd, 383 B/s wr, 7 op/s 2024-09-13T15:33:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:23 smithi090 ceph-mon[27811]: pgmap v242: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 383 B/s wr, 0 op/s 2024-09-13T15:33:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:23 smithi175 ceph-mon[36128]: pgmap v242: 129 pgs: 1 peering, 128 active+clean; 583 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 383 B/s wr, 0 op/s 2024-09-13T15:33:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:25 smithi090 ceph-mon[27811]: pgmap v243: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:25 smithi175 ceph-mon[36128]: pgmap v243: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:28.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:27 smithi090 ceph-mon[27811]: pgmap v244: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 639 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:27 smithi175 ceph-mon[36128]: pgmap v244: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 639 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:29.938 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:29 smithi090 ceph-mon[27811]: pgmap v245: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 615 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:29 smithi175 ceph-mon[36128]: pgmap v245: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 615 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:31 smithi090 ceph-mon[27811]: pgmap v246: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 558 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:31 smithi175 ceph-mon[36128]: pgmap v246: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 558 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:33 smithi090 ceph-mon[27811]: pgmap v247: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:33 smithi175 ceph-mon[36128]: pgmap v247: 129 pgs: 129 active+clean; 586 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:35 smithi090 ceph-mon[27811]: pgmap v248: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:36.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:35 smithi175 ceph-mon[36128]: pgmap v248: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-13T15:33:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:37 smithi090 ceph-mon[27811]: pgmap v249: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-13T15:33:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:37 smithi175 ceph-mon[36128]: pgmap v249: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-13T15:33:40.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:39 smithi090 ceph-mon[27811]: pgmap v250: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-13T15:33:40.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:39 smithi175 ceph-mon[36128]: pgmap v250: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-09-13T15:33:41.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:41 smithi175 ceph-mon[36128]: pgmap v251: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:33:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:41 smithi090 ceph-mon[27811]: pgmap v251: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:33:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:43 smithi090 ceph-mon[27811]: pgmap v252: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:33:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:43 smithi175 ceph-mon[36128]: pgmap v252: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:33:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:45 smithi090 ceph-mon[27811]: pgmap v253: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:33:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:45 smithi175 ceph-mon[36128]: pgmap v253: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:33:48.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:47 smithi090 ceph-mon[27811]: pgmap v254: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:48.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:47 smithi175 ceph-mon[36128]: pgmap v254: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:49.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:33:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:49 smithi090 ceph-mon[27811]: pgmap v255: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:49 smithi175 ceph-mon[36128]: pgmap v255: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:51 smithi090 ceph-mon[27811]: pgmap v256: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:51 smithi175 ceph-mon[36128]: pgmap v256: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:53 smithi090 ceph-mon[27811]: pgmap v257: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:54.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:53 smithi175 ceph-mon[36128]: pgmap v257: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:55.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:55 smithi090 ceph-mon[27811]: pgmap v258: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:55 smithi175 ceph-mon[36128]: pgmap v258: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:57 smithi090 ceph-mon[27811]: pgmap v259: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:33:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:57 smithi175 ceph-mon[36128]: pgmap v259: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:33:59 smithi090 ceph-mon[27811]: pgmap v260: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:33:59 smithi175 ceph-mon[36128]: pgmap v260: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:01 smithi090 ceph-mon[27811]: pgmap v261: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:01 smithi175 ceph-mon[36128]: pgmap v261: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:03 smithi090 ceph-mon[27811]: pgmap v262: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:03 smithi175 ceph-mon[36128]: pgmap v262: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:05 smithi090 ceph-mon[27811]: pgmap v263: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:06.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:05 smithi175 ceph-mon[36128]: pgmap v263: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:07 smithi090 ceph-mon[27811]: pgmap v264: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:08.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:07 smithi175 ceph-mon[36128]: pgmap v264: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:09 smithi090 ceph-mon[27811]: pgmap v265: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:09 smithi175 ceph-mon[36128]: pgmap v265: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:11 smithi090 ceph-mon[27811]: pgmap v266: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:12.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:11 smithi175 ceph-mon[36128]: pgmap v266: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:13 smithi090 ceph-mon[27811]: pgmap v267: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:14.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:13 smithi175 ceph-mon[36128]: pgmap v267: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:15.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:14 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:34:15.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:14 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:34:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:15 smithi090 ceph-mon[27811]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:34:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:34:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:34:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:15 smithi175 ceph-mon[36128]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:34:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:34:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:34:18.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:17 smithi090 ceph-mon[27811]: pgmap v269: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:17 smithi175 ceph-mon[36128]: pgmap v269: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:19 smithi090 ceph-mon[27811]: pgmap v270: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:19 smithi175 ceph-mon[36128]: pgmap v270: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:21 smithi090 ceph-mon[27811]: pgmap v271: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:22.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:21 smithi175 ceph-mon[36128]: pgmap v271: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:22.257 INFO:tasks.workunit.client.0.smithi090.stderr:Updating files: 97% (12349/12697) Updating files: 98% (12444/12697) Updating files: 99% (12571/12697) Updating files: 100% (12697/12697) Updating files: 100% (12697/12697), done. 2024-09-13T15:34:22.851 INFO:tasks.workunit.client.1.smithi175.stderr:Note: switching to '50bbe7f395997dcd871813949c49addbbf630361'. 2024-09-13T15:34:22.851 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.851 INFO:tasks.workunit.client.1.smithi175.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T15:34:22.851 INFO:tasks.workunit.client.1.smithi175.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T15:34:22.851 INFO:tasks.workunit.client.1.smithi175.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T15:34:22.852 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.852 INFO:tasks.workunit.client.1.smithi175.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T15:34:22.852 INFO:tasks.workunit.client.1.smithi175.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T15:34:22.852 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.852 INFO:tasks.workunit.client.1.smithi175.stderr: git switch -c 2024-09-13T15:34:22.852 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr:Or undo this operation with: 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr: git switch - 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr: 2024-09-13T15:34:22.853 INFO:tasks.workunit.client.1.smithi175.stderr:HEAD is now at 50bbe7f3959 Merge PR #58789 into wip-rishabh-testing-20240910.095512-reef 2024-09-13T15:34:22.857 DEBUG:teuthology.orchestra.run.smithi175:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-09-13T15:34:22.925 INFO:tasks.workunit.client.1.smithi175.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T15:34:22.927 INFO:tasks.workunit.client.1.smithi175.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-13T15:34:22.927 INFO:tasks.workunit.client.1.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T15:34:22.937 INFO:tasks.workunit.client.0.smithi090.stderr:Note: switching to '50bbe7f395997dcd871813949c49addbbf630361'. 2024-09-13T15:34:22.937 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr:state without impacting any branches by switching back to a branch. 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr: git switch -c 2024-09-13T15:34:22.938 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr:Or undo this operation with: 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr: git switch - 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr: 2024-09-13T15:34:22.939 INFO:tasks.workunit.client.0.smithi090.stderr:HEAD is now at 50bbe7f3959 Merge PR #58789 into wip-rishabh-testing-20240910.095512-reef 2024-09-13T15:34:22.944 DEBUG:teuthology.orchestra.run.smithi090:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-13T15:34:23.026 INFO:tasks.workunit.client.0.smithi090.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-13T15:34:23.029 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:34:23.029 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-13T15:34:23.592 INFO:tasks.workunit.client.1.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T15:34:23.674 INFO:tasks.workunit.client.1.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T15:34:23.716 INFO:tasks.workunit.client.1.smithi175.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-09-13T15:34:23.718 INFO:tasks.workunit.client.1.smithi175.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-13T15:34:23.718 INFO:tasks.workunit.client.1.smithi175.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T15:34:23.757 INFO:tasks.workunit.client.1.smithi175.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-09-13T15:34:23.761 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:34:23.761 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-09-13T15:34:23.794 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-13T15:34:23.817 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.1... 2024-09-13T15:34:23.818 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-13T15:34:23.819 DEBUG:teuthology.orchestra.run.smithi175:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bbe7f395997dcd871813949c49addbbf630361 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh 2024-09-13T15:34:23.890 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-13T15:34:23.894 INFO:tasks.workunit.client.1.smithi175.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-13T15:34:23.894 INFO:tasks.workunit.client.1.smithi175.stdout: 2024-09-13T15:34:23.894 INFO:tasks.workunit.client.1.smithi175.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-13T15:34:23.901 INFO:tasks.workunit.client.1.smithi175.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-13T15:34:23.931 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-13T15:34:23.933 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:34:23.933 INFO:tasks.workunit.client.0.smithi090.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-13T15:34:23.972 INFO:tasks.workunit.client.0.smithi090.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-13T15:34:23.975 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:34:23.975 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-13T15:34:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:23 smithi090 ceph-mon[27811]: pgmap v272: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:24.002 INFO:tasks.workunit.client.1.smithi175.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-13T15:34:24.002 INFO:tasks.workunit.client.1.smithi175.stdout:releasing clients 2024-09-13T15:34:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:23 smithi175 ceph-mon[36128]: pgmap v272: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:24.017 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-09-13T15:34:24.020 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-09-13T15:34:24.020 DEBUG:teuthology.orchestra.run.smithi090:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50bbe7f395997dcd871813949c49addbbf630361 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-09-13T15:34:24.090 INFO:tasks.workunit.client.0.smithi090.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-13T15:34:24.090 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-09-13T15:34:24.090 INFO:tasks.workunit.client.0.smithi090.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-13T15:34:24.094 INFO:tasks.workunit.client.0.smithi090.stdout:0 of 1 processes prepared for launch 0 sec 2024-09-13T15:34:24.194 INFO:tasks.workunit.client.0.smithi090.stdout:1 of 1 processes prepared for launch 0 sec 2024-09-13T15:34:24.194 INFO:tasks.workunit.client.0.smithi090.stdout:releasing clients 2024-09-13T15:34:25.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 1063 22.89 MB/sec warmup 1 sec latency 10.858 ms 2024-09-13T15:34:25.194 INFO:tasks.workunit.client.0.smithi090.stdout: 1 2768 31.97 MB/sec warmup 1 sec latency 101.179 ms 2024-09-13T15:34:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:25 smithi090 ceph-mon[27811]: pgmap v273: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:26.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 1963 13.23 MB/sec warmup 2 sec latency 14.107 ms 2024-09-13T15:34:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:25 smithi175 ceph-mon[36128]: pgmap v273: 129 pgs: 129 active+clean; 587 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-09-13T15:34:26.194 INFO:tasks.workunit.client.0.smithi090.stdout: 1 5451 22.75 MB/sec warmup 2 sec latency 74.559 ms 2024-09-13T15:34:27.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 2892 10.91 MB/sec warmup 3 sec latency 11.506 ms 2024-09-13T15:34:27.194 INFO:tasks.workunit.client.0.smithi090.stdout: 1 8094 20.05 MB/sec warmup 3 sec latency 33.430 ms 2024-09-13T15:34:27.780 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:27 smithi090 ceph-mon[27811]: pgmap v274: 129 pgs: 129 active+clean; 4.3 MiB data, 241 MiB used, 715 GiB / 715 GiB avail; 324 KiB/s wr, 1 op/s 2024-09-13T15:34:28.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 3676 9.68 MB/sec warmup 4 sec latency 16.070 ms 2024-09-13T15:34:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:27 smithi175 ceph-mon[36128]: pgmap v274: 129 pgs: 129 active+clean; 4.3 MiB data, 241 MiB used, 715 GiB / 715 GiB avail; 324 KiB/s wr, 1 op/s 2024-09-13T15:34:28.194 INFO:tasks.workunit.client.0.smithi090.stdout: 1 11532 19.75 MB/sec warmup 4 sec latency 52.324 ms 2024-09-13T15:34:29.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 4243 8.06 MB/sec warmup 5 sec latency 13.057 ms 2024-09-13T15:34:29.194 INFO:tasks.workunit.client.0.smithi090.stdout: 1 14940 19.56 MB/sec warmup 5 sec latency 59.967 ms 2024-09-13T15:34:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:29 smithi090 ceph-mon[27811]: pgmap v275: 129 pgs: 129 active+clean; 21 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.0 MiB/s wr, 30 op/s 2024-09-13T15:34:30.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 4963 7.52 MB/sec warmup 6 sec latency 129.855 ms 2024-09-13T15:34:30.018 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:29 smithi175 ceph-mon[36128]: pgmap v275: 129 pgs: 129 active+clean; 21 MiB data, 293 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.0 MiB/s wr, 30 op/s 2024-09-13T15:34:30.194 INFO:tasks.workunit.client.0.smithi090.stdout: 1 17489 18.64 MB/sec warmup 6 sec latency 318.486 ms 2024-09-13T15:34:31.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 5867 6.70 MB/sec warmup 7 sec latency 10.837 ms 2024-09-13T15:34:31.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 20702 18.45 MB/sec warmup 7 sec latency 31.932 ms 2024-09-13T15:34:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:31 smithi090 ceph-mon[27811]: pgmap v276: 129 pgs: 129 active+clean; 24 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.2 MiB/s wr, 33 op/s 2024-09-13T15:34:32.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 6846 6.81 MB/sec warmup 8 sec latency 12.979 ms 2024-09-13T15:34:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:31 smithi175 ceph-mon[36128]: pgmap v276: 129 pgs: 129 active+clean; 24 MiB data, 303 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.2 MiB/s wr, 33 op/s 2024-09-13T15:34:32.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 24323 18.54 MB/sec warmup 8 sec latency 32.315 ms 2024-09-13T15:34:33.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 7458 6.58 MB/sec warmup 9 sec latency 9.981 ms 2024-09-13T15:34:33.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 27292 18.00 MB/sec warmup 9 sec latency 59.839 ms 2024-09-13T15:34:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:33 smithi090 ceph-mon[27811]: pgmap v277: 129 pgs: 129 active+clean; 24 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.4 MiB/s wr, 38 op/s 2024-09-13T15:34:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:34.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 8104 6.01 MB/sec warmup 10 sec latency 14.095 ms 2024-09-13T15:34:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:33 smithi175 ceph-mon[36128]: pgmap v277: 129 pgs: 129 active+clean; 24 MiB data, 308 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 2.4 MiB/s wr, 38 op/s 2024-09-13T15:34:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:34.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 30764 18.09 MB/sec warmup 10 sec latency 30.854 ms 2024-09-13T15:34:35.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 8947 5.87 MB/sec warmup 11 sec latency 46.923 ms 2024-09-13T15:34:35.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 32704 17.59 MB/sec warmup 11 sec latency 303.066 ms 2024-09-13T15:34:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:35 smithi090 ceph-mon[27811]: pgmap v278: 129 pgs: 129 active+clean; 25 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 3.0 MiB/s wr, 72 op/s 2024-09-13T15:34:36.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 9593 5.58 MB/sec warmup 12 sec latency 114.098 ms 2024-09-13T15:34:36.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:35 smithi175 ceph-mon[36128]: pgmap v278: 129 pgs: 129 active+clean; 25 MiB data, 316 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 3.0 MiB/s wr, 72 op/s 2024-09-13T15:34:36.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 32765 16.13 MB/sec warmup 12 sec latency 370.796 ms 2024-09-13T15:34:37.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 10308 5.58 MB/sec warmup 13 sec latency 142.261 ms 2024-09-13T15:34:37.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 32772 14.89 MB/sec warmup 13 sec latency 434.455 ms 2024-09-13T15:34:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:37 smithi090 ceph-mon[27811]: pgmap v279: 129 pgs: 129 active+clean; 25 MiB data, 317 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 3.1 MiB/s wr, 75 op/s 2024-09-13T15:34:38.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 10882 5.57 MB/sec warmup 14 sec latency 67.625 ms 2024-09-13T15:34:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:37 smithi175 ceph-mon[36128]: pgmap v279: 129 pgs: 129 active+clean; 25 MiB data, 317 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 3.1 MiB/s wr, 75 op/s 2024-09-13T15:34:38.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 32848 13.84 MB/sec warmup 14 sec latency 341.198 ms 2024-09-13T15:34:39.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 11309 5.23 MB/sec warmup 15 sec latency 153.578 ms 2024-09-13T15:34:39.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 34311 13.34 MB/sec warmup 15 sec latency 393.024 ms 2024-09-13T15:34:40.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:39 smithi090 ceph-mon[27811]: pgmap v280: 129 pgs: 129 active+clean; 26 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.3 MiB/s wr, 103 op/s 2024-09-13T15:34:40.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 12174 5.21 MB/sec warmup 16 sec latency 14.693 ms 2024-09-13T15:34:40.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:39 smithi175 ceph-mon[36128]: pgmap v280: 129 pgs: 129 active+clean; 26 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.3 MiB/s wr, 103 op/s 2024-09-13T15:34:40.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 36435 13.31 MB/sec warmup 16 sec latency 142.049 ms 2024-09-13T15:34:41.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 13015 5.02 MB/sec warmup 17 sec latency 14.079 ms 2024-09-13T15:34:41.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 39790 13.63 MB/sec warmup 17 sec latency 172.481 ms 2024-09-13T15:34:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:41 smithi090 ceph-mon[27811]: pgmap v281: 129 pgs: 129 active+clean; 26 MiB data, 324 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:34:42.001 INFO:tasks.workunit.client.1.smithi175.stdout: 1 13956 5.15 MB/sec warmup 18 sec latency 12.367 ms 2024-09-13T15:34:42.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:41 smithi175 ceph-mon[36128]: pgmap v281: 129 pgs: 129 active+clean; 26 MiB data, 324 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:34:42.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 39856 12.88 MB/sec warmup 18 sec latency 272.203 ms 2024-09-13T15:34:43.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 14576 5.13 MB/sec warmup 19 sec latency 16.891 ms 2024-09-13T15:34:43.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 39863 12.20 MB/sec warmup 19 sec latency 278.778 ms 2024-09-13T15:34:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:43 smithi090 ceph-mon[27811]: pgmap v282: 129 pgs: 129 active+clean; 26 MiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-13T15:34:44.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 15265 4.97 MB/sec warmup 20 sec latency 13.065 ms 2024-09-13T15:34:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:43 smithi175 ceph-mon[36128]: pgmap v282: 129 pgs: 129 active+clean; 26 MiB data, 325 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 79 op/s 2024-09-13T15:34:44.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 39939 11.60 MB/sec warmup 20 sec latency 338.125 ms 2024-09-13T15:34:45.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 15971 4.89 MB/sec warmup 21 sec latency 131.704 ms 2024-09-13T15:34:45.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 39982 11.05 MB/sec warmup 21 sec latency 404.886 ms 2024-09-13T15:34:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:45 smithi090 ceph-mon[27811]: pgmap v283: 129 pgs: 129 active+clean; 28 MiB data, 333 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:34:46.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 16554 4.74 MB/sec warmup 22 sec latency 146.748 ms 2024-09-13T15:34:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:45 smithi175 ceph-mon[36128]: pgmap v283: 129 pgs: 129 active+clean; 28 MiB data, 333 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:34:46.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 40696 10.75 MB/sec warmup 22 sec latency 371.609 ms 2024-09-13T15:34:47.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 17504 4.86 MB/sec warmup 23 sec latency 10.644 ms 2024-09-13T15:34:47.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 43587 10.93 MB/sec warmup 23 sec latency 42.783 ms 2024-09-13T15:34:47.946 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:47 smithi175 ceph-mon[36128]: pgmap v284: 129 pgs: 129 active+clean; 28 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-13T15:34:48.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:47 smithi090 ceph-mon[27811]: pgmap v284: 129 pgs: 129 active+clean; 28 MiB data, 337 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-13T15:34:48.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 18144 4.86 MB/sec warmup 24 sec latency 10.388 ms 2024-09-13T15:34:48.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 47031 11.26 MB/sec warmup 24 sec latency 36.163 ms 2024-09-13T15:34:49.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:49.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 18847 4.74 MB/sec warmup 25 sec latency 10.706 ms 2024-09-13T15:34:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:34:49.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 50498 11.57 MB/sec warmup 25 sec latency 32.297 ms 2024-09-13T15:34:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:49 smithi090 ceph-mon[27811]: pgmap v285: 129 pgs: 129 active+clean; 28 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-09-13T15:34:50.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 19748 4.70 MB/sec warmup 26 sec latency 9.848 ms 2024-09-13T15:34:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:49 smithi175 ceph-mon[36128]: pgmap v285: 129 pgs: 129 active+clean; 28 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-09-13T15:34:50.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 54039 11.86 MB/sec warmup 26 sec latency 32.307 ms 2024-09-13T15:34:51.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 20717 4.78 MB/sec warmup 27 sec latency 10.388 ms 2024-09-13T15:34:51.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 57518 12.12 MB/sec warmup 27 sec latency 54.840 ms 2024-09-13T15:34:52.004 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:51 smithi090 ceph-mon[27811]: pgmap v286: 129 pgs: 129 active+clean; 29 MiB data, 343 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:34:52.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 21387 4.79 MB/sec warmup 28 sec latency 137.936 ms 2024-09-13T15:34:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:51 smithi175 ceph-mon[36128]: pgmap v286: 129 pgs: 129 active+clean; 29 MiB data, 343 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:34:52.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 57525 11.69 MB/sec warmup 28 sec latency 363.450 ms 2024-09-13T15:34:53.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 21801 4.68 MB/sec warmup 29 sec latency 98.218 ms 2024-09-13T15:34:53.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 57586 11.29 MB/sec warmup 29 sec latency 367.853 ms 2024-09-13T15:34:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:53 smithi090 ceph-mon[27811]: pgmap v287: 129 pgs: 129 active+clean; 29 MiB data, 346 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:34:54.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 22506 4.59 MB/sec warmup 30 sec latency 55.899 ms 2024-09-13T15:34:54.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:53 smithi175 ceph-mon[36128]: pgmap v287: 129 pgs: 129 active+clean; 29 MiB data, 346 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:34:54.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 61007 11.54 MB/sec warmup 30 sec latency 375.670 ms 2024-09-13T15:34:55.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 23388 4.56 MB/sec warmup 31 sec latency 15.557 ms 2024-09-13T15:34:55.195 INFO:tasks.workunit.client.0.smithi090.stdout: 1 63630 11.63 MB/sec warmup 31 sec latency 33.377 ms 2024-09-13T15:34:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:55 smithi090 ceph-mon[27811]: pgmap v288: 129 pgs: 129 active+clean; 31 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 108 op/s 2024-09-13T15:34:56.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 24290 4.63 MB/sec warmup 32 sec latency 67.008 ms 2024-09-13T15:34:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:55 smithi175 ceph-mon[36128]: pgmap v288: 129 pgs: 129 active+clean; 31 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 108 op/s 2024-09-13T15:34:56.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 66288 11.63 MB/sec warmup 32 sec latency 127.439 ms 2024-09-13T15:34:57.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 25088 4.68 MB/sec warmup 33 sec latency 11.440 ms 2024-09-13T15:34:57.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 69058 11.80 MB/sec warmup 33 sec latency 65.484 ms 2024-09-13T15:34:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:57 smithi090 ceph-mon[27811]: pgmap v289: 129 pgs: 129 active+clean; 31 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:34:58.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 25666 4.57 MB/sec warmup 34 sec latency 10.537 ms 2024-09-13T15:34:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:57 smithi175 ceph-mon[36128]: pgmap v289: 129 pgs: 129 active+clean; 31 MiB data, 355 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:34:58.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 71886 11.89 MB/sec warmup 34 sec latency 50.957 ms 2024-09-13T15:34:59.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 26386 4.57 MB/sec warmup 35 sec latency 120.731 ms 2024-09-13T15:34:59.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 72856 11.68 MB/sec warmup 35 sec latency 358.717 ms 2024-09-13T15:35:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:34:59 smithi090 ceph-mon[27811]: pgmap v290: 129 pgs: 129 active+clean; 33 MiB data, 362 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 107 op/s 2024-09-13T15:35:00.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 27068 4.46 MB/sec warmup 36 sec latency 132.839 ms 2024-09-13T15:35:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:34:59 smithi175 ceph-mon[36128]: pgmap v290: 129 pgs: 129 active+clean; 33 MiB data, 362 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 107 op/s 2024-09-13T15:35:00.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 73343 11.40 MB/sec warmup 36 sec latency 362.902 ms 2024-09-13T15:35:01.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 28138 4.57 MB/sec warmup 37 sec latency 11.500 ms 2024-09-13T15:35:01.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 75470 11.44 MB/sec warmup 37 sec latency 367.910 ms 2024-09-13T15:35:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:01 smithi090 ceph-mon[27811]: pgmap v291: 129 pgs: 129 active+clean; 33 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:35:02.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 28719 4.57 MB/sec warmup 38 sec latency 13.644 ms 2024-09-13T15:35:02.018 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:01 smithi175 ceph-mon[36128]: pgmap v291: 129 pgs: 129 active+clean; 33 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:35:02.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 78970 11.64 MB/sec warmup 38 sec latency 33.268 ms 2024-09-13T15:35:03.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 29354 4.48 MB/sec warmup 39 sec latency 11.066 ms 2024-09-13T15:35:03.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 82390 11.83 MB/sec warmup 39 sec latency 57.408 ms 2024-09-13T15:35:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:03 smithi090 ceph-mon[27811]: pgmap v292: 129 pgs: 129 active+clean; 32 MiB data, 362 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:35:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:04.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 30222 4.48 MB/sec warmup 40 sec latency 13.197 ms 2024-09-13T15:35:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:03 smithi175 ceph-mon[36128]: pgmap v292: 129 pgs: 129 active+clean; 32 MiB data, 362 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:35:04.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:04.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 85858 12.00 MB/sec warmup 40 sec latency 158.436 ms 2024-09-13T15:35:05.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 30999 4.44 MB/sec warmup 41 sec latency 11.429 ms 2024-09-13T15:35:05.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 88480 12.06 MB/sec warmup 41 sec latency 33.529 ms 2024-09-13T15:35:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:05 smithi090 ceph-mon[27811]: pgmap v293: 129 pgs: 129 active+clean; 34 MiB data, 376 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 108 op/s 2024-09-13T15:35:06.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 31751 4.49 MB/sec warmup 42 sec latency 103.651 ms 2024-09-13T15:35:06.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:05 smithi175 ceph-mon[36128]: pgmap v293: 129 pgs: 129 active+clean; 34 MiB data, 376 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 108 op/s 2024-09-13T15:35:06.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 89467 11.89 MB/sec warmup 42 sec latency 381.427 ms 2024-09-13T15:35:07.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 32348 4.49 MB/sec warmup 43 sec latency 121.797 ms 2024-09-13T15:35:07.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 89528 11.62 MB/sec warmup 43 sec latency 368.224 ms 2024-09-13T15:35:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:07 smithi090 ceph-mon[27811]: pgmap v294: 129 pgs: 129 active+clean; 34 MiB data, 376 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-13T15:35:08.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 32968 4.41 MB/sec warmup 44 sec latency 72.763 ms 2024-09-13T15:35:08.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:07 smithi175 ceph-mon[36128]: pgmap v294: 129 pgs: 129 active+clean; 34 MiB data, 376 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-13T15:35:08.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 91645 11.63 MB/sec warmup 44 sec latency 351.087 ms 2024-09-13T15:35:09.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 33838 4.41 MB/sec warmup 45 sec latency 13.460 ms 2024-09-13T15:35:09.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 94163 11.64 MB/sec warmup 45 sec latency 82.349 ms 2024-09-13T15:35:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:09 smithi090 ceph-mon[27811]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 379 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 109 op/s 2024-09-13T15:35:10.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 34724 4.44 MB/sec warmup 46 sec latency 15.500 ms 2024-09-13T15:35:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:09 smithi175 ceph-mon[36128]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 379 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 109 op/s 2024-09-13T15:35:10.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 97304 11.79 MB/sec warmup 46 sec latency 35.556 ms 2024-09-13T15:35:11.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 35625 4.49 MB/sec warmup 47 sec latency 11.672 ms 2024-09-13T15:35:11.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 100174 11.85 MB/sec warmup 47 sec latency 116.710 ms 2024-09-13T15:35:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:11 smithi090 ceph-mon[27811]: pgmap v296: 129 pgs: 129 active+clean; 36 MiB data, 380 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:35:12.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 36227 4.43 MB/sec warmup 48 sec latency 9.735 ms 2024-09-13T15:35:12.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:11 smithi175 ceph-mon[36128]: pgmap v296: 129 pgs: 129 active+clean; 36 MiB data, 380 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:35:12.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 103712 12.00 MB/sec warmup 48 sec latency 30.401 ms 2024-09-13T15:35:13.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 37000 4.43 MB/sec warmup 49 sec latency 131.540 ms 2024-09-13T15:35:13.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 105328 11.89 MB/sec warmup 49 sec latency 337.821 ms 2024-09-13T15:35:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:13 smithi090 ceph-mon[27811]: pgmap v297: 129 pgs: 129 active+clean; 36 MiB data, 379 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:35:14.002 INFO:tasks.workunit.client.1.smithi175.stdout: 1 37657 4.36 MB/sec warmup 50 sec latency 138.583 ms 2024-09-13T15:35:14.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:13 smithi175 ceph-mon[36128]: pgmap v297: 129 pgs: 129 active+clean; 36 MiB data, 379 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:35:14.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 105829 11.76 MB/sec warmup 50 sec latency 358.220 ms 2024-09-13T15:35:15.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 38494 4.41 MB/sec warmup 51 sec latency 63.258 ms 2024-09-13T15:35:15.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 107129 11.66 MB/sec warmup 51 sec latency 424.444 ms 2024-09-13T15:35:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:15 smithi090 ceph-mon[27811]: pgmap v298: 129 pgs: 129 active+clean; 36 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:35:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:35:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:35:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:35:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:35:16.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 39279 4.44 MB/sec warmup 52 sec latency 12.339 ms 2024-09-13T15:35:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:15 smithi175 ceph-mon[36128]: pgmap v298: 129 pgs: 129 active+clean; 36 MiB data, 386 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:35:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:35:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:35:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:35:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:35:16.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 109755 11.71 MB/sec warmup 52 sec latency 33.286 ms 2024-09-13T15:35:17.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 39843 4.37 MB/sec warmup 53 sec latency 10.479 ms 2024-09-13T15:35:17.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 112918 11.82 MB/sec warmup 53 sec latency 33.327 ms 2024-09-13T15:35:18.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:17 smithi090 ceph-mon[27811]: pgmap v299: 129 pgs: 129 active+clean; 36 MiB data, 387 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:35:18.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 40672 4.38 MB/sec warmup 54 sec latency 10.383 ms 2024-09-13T15:35:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:17 smithi175 ceph-mon[36128]: pgmap v299: 129 pgs: 129 active+clean; 36 MiB data, 387 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:35:18.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 115921 11.85 MB/sec warmup 54 sec latency 34.940 ms 2024-09-13T15:35:18.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:19.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 41487 4.34 MB/sec warmup 55 sec latency 12.766 ms 2024-09-13T15:35:19.196 INFO:tasks.workunit.client.0.smithi090.stdout: 1 118980 11.95 MB/sec warmup 55 sec latency 33.332 ms 2024-09-13T15:35:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:19 smithi090 ceph-mon[27811]: pgmap v300: 129 pgs: 129 active+clean; 38 MiB data, 398 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 104 op/s 2024-09-13T15:35:20.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 42379 4.39 MB/sec warmup 56 sec latency 22.355 ms 2024-09-13T15:35:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:19 smithi175 ceph-mon[36128]: pgmap v300: 129 pgs: 129 active+clean; 38 MiB data, 398 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 104 op/s 2024-09-13T15:35:20.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 121317 11.99 MB/sec warmup 56 sec latency 248.577 ms 2024-09-13T15:35:21.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 42981 4.39 MB/sec warmup 57 sec latency 132.600 ms 2024-09-13T15:35:21.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 121366 11.78 MB/sec warmup 57 sec latency 358.494 ms 2024-09-13T15:35:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:21 smithi090 ceph-mon[27811]: pgmap v301: 129 pgs: 129 active+clean; 38 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:35:22.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 43422 4.33 MB/sec warmup 58 sec latency 129.933 ms 2024-09-13T15:35:22.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:21 smithi175 ceph-mon[36128]: pgmap v301: 129 pgs: 129 active+clean; 38 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:35:22.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 121441 11.58 MB/sec warmup 58 sec latency 359.547 ms 2024-09-13T15:35:23.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 44244 4.33 MB/sec warmup 59 sec latency 14.710 ms 2024-09-13T15:35:23.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 124927 11.71 MB/sec warmup 59 sec latency 43.627 ms 2024-09-13T15:35:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:23 smithi090 ceph-mon[27811]: pgmap v302: 129 pgs: 129 active+clean; 39 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-13T15:35:24.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 45053 4.29 MB/sec warmup 60 sec latency 12.431 ms 2024-09-13T15:35:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:23 smithi175 ceph-mon[36128]: pgmap v302: 129 pgs: 129 active+clean; 39 MiB data, 405 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 97 op/s 2024-09-13T15:35:24.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 128415 11.83 MB/sec warmup 60 sec latency 74.934 ms 2024-09-13T15:35:25.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 45931 4.34 MB/sec warmup 61 sec latency 12.202 ms 2024-09-13T15:35:25.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 131055 11.87 MB/sec warmup 61 sec latency 75.201 ms 2024-09-13T15:35:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:25 smithi090 ceph-mon[27811]: pgmap v303: 129 pgs: 129 active+clean; 39 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-09-13T15:35:26.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 46622 4.35 MB/sec warmup 62 sec latency 11.996 ms 2024-09-13T15:35:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:25 smithi175 ceph-mon[36128]: pgmap v303: 129 pgs: 129 active+clean; 39 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 94 op/s 2024-09-13T15:35:26.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 133756 11.88 MB/sec warmup 62 sec latency 122.518 ms 2024-09-13T15:35:27.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 47379 4.31 MB/sec warmup 63 sec latency 12.403 ms 2024-09-13T15:35:27.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 137167 11.98 MB/sec warmup 63 sec latency 33.284 ms 2024-09-13T15:35:27.781 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:27 smithi090 ceph-mon[27811]: pgmap v304: 129 pgs: 129 active+clean; 39 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:35:28.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 48290 4.30 MB/sec warmup 64 sec latency 11.407 ms 2024-09-13T15:35:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:27 smithi175 ceph-mon[36128]: pgmap v304: 129 pgs: 129 active+clean; 39 MiB data, 395 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:35:28.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 140005 12.06 MB/sec warmup 64 sec latency 41.826 ms 2024-09-13T15:35:29.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 49310 4.35 MB/sec warmup 65 sec latency 15.504 ms 2024-09-13T15:35:29.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 142838 12.10 MB/sec warmup 65 sec latency 33.445 ms 2024-09-13T15:35:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:29 smithi090 ceph-mon[27811]: pgmap v305: 129 pgs: 129 active+clean; 40 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-09-13T15:35:30.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 49967 4.37 MB/sec warmup 66 sec latency 11.589 ms 2024-09-13T15:35:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:29 smithi175 ceph-mon[36128]: pgmap v305: 129 pgs: 129 active+clean; 40 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-09-13T15:35:30.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 146272 12.21 MB/sec warmup 66 sec latency 33.200 ms 2024-09-13T15:35:31.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 50593 4.32 MB/sec warmup 67 sec latency 10.583 ms 2024-09-13T15:35:31.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 149745 12.31 MB/sec warmup 67 sec latency 50.038 ms 2024-09-13T15:35:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:31 smithi090 ceph-mon[27811]: pgmap v306: 129 pgs: 129 active+clean; 41 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-13T15:35:32.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 51468 4.32 MB/sec warmup 68 sec latency 13.741 ms 2024-09-13T15:35:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:31 smithi175 ceph-mon[36128]: pgmap v306: 129 pgs: 129 active+clean; 41 MiB data, 403 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-13T15:35:32.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 153283 12.41 MB/sec warmup 68 sec latency 51.992 ms 2024-09-13T15:35:33.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 52241 4.30 MB/sec warmup 69 sec latency 17.541 ms 2024-09-13T15:35:33.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 156776 12.50 MB/sec warmup 69 sec latency 41.573 ms 2024-09-13T15:35:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:33 smithi090 ceph-mon[27811]: pgmap v307: 129 pgs: 129 active+clean; 42 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-13T15:35:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:34.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 53080 4.33 MB/sec warmup 70 sec latency 10.590 ms 2024-09-13T15:35:34.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:33 smithi175 ceph-mon[36128]: pgmap v307: 129 pgs: 129 active+clean; 42 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-13T15:35:34.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:34.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 159419 12.53 MB/sec warmup 70 sec latency 40.591 ms 2024-09-13T15:35:35.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 53763 4.34 MB/sec warmup 71 sec latency 10.568 ms 2024-09-13T15:35:35.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 162639 12.59 MB/sec warmup 71 sec latency 33.147 ms 2024-09-13T15:35:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:35 smithi090 ceph-mon[27811]: pgmap v308: 129 pgs: 129 active+clean; 42 MiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-13T15:35:36.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 54625 4.35 MB/sec warmup 72 sec latency 8.803 ms 2024-09-13T15:35:36.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:35 smithi175 ceph-mon[36128]: pgmap v308: 129 pgs: 129 active+clean; 42 MiB data, 411 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-13T15:35:36.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 165661 12.62 MB/sec warmup 72 sec latency 33.450 ms 2024-09-13T15:35:37.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 55497 4.30 MB/sec warmup 73 sec latency 11.602 ms 2024-09-13T15:35:37.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 168623 12.67 MB/sec warmup 73 sec latency 62.716 ms 2024-09-13T15:35:38.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 56519 4.35 MB/sec warmup 74 sec latency 12.578 ms 2024-09-13T15:35:38.013 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:37 smithi090 ceph-mon[27811]: pgmap v309: 129 pgs: 129 active+clean; 41 MiB data, 412 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:35:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:37 smithi175 ceph-mon[36128]: pgmap v309: 129 pgs: 129 active+clean; 41 MiB data, 412 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:35:38.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 171761 12.76 MB/sec warmup 74 sec latency 33.243 ms 2024-09-13T15:35:39.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 57120 4.36 MB/sec warmup 75 sec latency 11.899 ms 2024-09-13T15:35:39.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 174709 12.78 MB/sec warmup 75 sec latency 33.396 ms 2024-09-13T15:35:40.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 57727 4.31 MB/sec warmup 76 sec latency 54.226 ms 2024-09-13T15:35:40.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:39 smithi175 ceph-mon[36128]: pgmap v310: 129 pgs: 129 active+clean; 44 MiB data, 422 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-09-13T15:35:40.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 178100 12.86 MB/sec warmup 76 sec latency 229.026 ms 2024-09-13T15:35:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:39 smithi090 ceph-mon[27811]: pgmap v310: 129 pgs: 129 active+clean; 44 MiB data, 422 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.4 MiB/s wr, 124 op/s 2024-09-13T15:35:41.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 58625 4.32 MB/sec warmup 77 sec latency 11.260 ms 2024-09-13T15:35:41.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 181596 12.94 MB/sec warmup 77 sec latency 33.416 ms 2024-09-13T15:35:42.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 59400 4.30 MB/sec warmup 78 sec latency 17.098 ms 2024-09-13T15:35:42.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:41 smithi175 ceph-mon[36128]: pgmap v311: 129 pgs: 129 active+clean; 44 MiB data, 423 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:35:42.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 184215 12.96 MB/sec warmup 78 sec latency 79.051 ms 2024-09-13T15:35:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:41 smithi090 ceph-mon[27811]: pgmap v311: 129 pgs: 129 active+clean; 44 MiB data, 423 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:35:43.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 60288 4.36 MB/sec warmup 79 sec latency 94.298 ms 2024-09-13T15:35:43.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 185382 12.86 MB/sec warmup 79 sec latency 314.864 ms 2024-09-13T15:35:44.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 60908 4.33 MB/sec warmup 80 sec latency 8.994 ms 2024-09-13T15:35:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:43 smithi175 ceph-mon[36128]: pgmap v312: 129 pgs: 129 active+clean; 42 MiB data, 425 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-13T15:35:44.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 188806 12.94 MB/sec warmup 80 sec latency 137.513 ms 2024-09-13T15:35:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:43 smithi090 ceph-mon[27811]: pgmap v312: 129 pgs: 129 active+clean; 42 MiB data, 425 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 90 op/s 2024-09-13T15:35:45.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 61764 4.33 MB/sec warmup 81 sec latency 9.703 ms 2024-09-13T15:35:45.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 192281 13.01 MB/sec warmup 81 sec latency 39.815 ms 2024-09-13T15:35:46.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 62671 4.30 MB/sec warmup 82 sec latency 10.959 ms 2024-09-13T15:35:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:45 smithi175 ceph-mon[36128]: pgmap v313: 129 pgs: 129 active+clean; 44 MiB data, 418 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-09-13T15:35:46.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 194851 13.02 MB/sec warmup 82 sec latency 184.649 ms 2024-09-13T15:35:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:45 smithi090 ceph-mon[27811]: pgmap v313: 129 pgs: 129 active+clean; 44 MiB data, 418 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.1 MiB/s wr, 114 op/s 2024-09-13T15:35:47.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 63634 4.34 MB/sec warmup 83 sec latency 15.672 ms 2024-09-13T15:35:47.197 INFO:tasks.workunit.client.0.smithi090.stdout: 1 197574 13.02 MB/sec warmup 83 sec latency 118.974 ms 2024-09-13T15:35:48.003 INFO:tasks.workunit.client.1.smithi175.stdout: 1 64272 4.35 MB/sec warmup 84 sec latency 11.715 ms 2024-09-13T15:35:48.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:47 smithi175 ceph-mon[36128]: pgmap v314: 129 pgs: 129 active+clean; 44 MiB data, 419 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:35:48.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 199565 13.01 MB/sec warmup 84 sec latency 207.034 ms 2024-09-13T15:35:48.232 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:47 smithi090 ceph-mon[27811]: pgmap v314: 129 pgs: 129 active+clean; 44 MiB data, 419 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:35:49.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 64794 4.31 MB/sec warmup 85 sec latency 115.690 ms 2024-09-13T15:35:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:49.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 200127 12.90 MB/sec warmup 85 sec latency 389.931 ms 2024-09-13T15:35:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:35:50.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 65486 4.31 MB/sec warmup 86 sec latency 101.820 ms 2024-09-13T15:35:50.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 200530 12.76 MB/sec warmup 86 sec latency 357.255 ms 2024-09-13T15:35:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:49 smithi090 ceph-mon[27811]: pgmap v315: 129 pgs: 129 active+clean; 46 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.4 MiB/s wr, 115 op/s 2024-09-13T15:35:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:49 smithi175 ceph-mon[36128]: pgmap v315: 129 pgs: 129 active+clean; 46 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.4 MiB/s wr, 115 op/s 2024-09-13T15:35:51.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 66046 4.26 MB/sec warmup 87 sec latency 139.067 ms 2024-09-13T15:35:51.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 200986 12.63 MB/sec warmup 87 sec latency 361.455 ms 2024-09-13T15:35:52.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 66649 4.27 MB/sec warmup 88 sec latency 140.809 ms 2024-09-13T15:35:52.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 201057 12.49 MB/sec warmup 88 sec latency 368.963 ms 2024-09-13T15:35:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:51 smithi090 ceph-mon[27811]: pgmap v316: 129 pgs: 129 active+clean; 46 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:35:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:51 smithi175 ceph-mon[36128]: pgmap v316: 129 pgs: 129 active+clean; 46 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:35:53.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 67205 4.26 MB/sec warmup 89 sec latency 123.155 ms 2024-09-13T15:35:53.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 201558 12.41 MB/sec warmup 89 sec latency 380.185 ms 2024-09-13T15:35:54.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 67775 4.27 MB/sec warmup 90 sec latency 135.726 ms 2024-09-13T15:35:54.133 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:53 smithi175 ceph-mon[36128]: pgmap v317: 129 pgs: 129 active+clean; 46 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 86 op/s 2024-09-13T15:35:54.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 201833 12.28 MB/sec warmup 90 sec latency 415.087 ms 2024-09-13T15:35:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:53 smithi090 ceph-mon[27811]: pgmap v317: 129 pgs: 129 active+clean; 46 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 86 op/s 2024-09-13T15:35:55.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 68201 4.23 MB/sec warmup 91 sec latency 141.181 ms 2024-09-13T15:35:55.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 201941 12.16 MB/sec warmup 91 sec latency 360.300 ms 2024-09-13T15:35:56.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 68711 4.20 MB/sec warmup 92 sec latency 158.719 ms 2024-09-13T15:35:56.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 202862 12.08 MB/sec warmup 92 sec latency 485.202 ms 2024-09-13T15:35:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:55 smithi090 ceph-mon[27811]: pgmap v318: 129 pgs: 129 active+clean; 47 MiB data, 422 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 105 op/s 2024-09-13T15:35:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:55 smithi175 ceph-mon[36128]: pgmap v318: 129 pgs: 129 active+clean; 47 MiB data, 422 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 105 op/s 2024-09-13T15:35:57.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 69484 4.19 MB/sec warmup 93 sec latency 74.092 ms 2024-09-13T15:35:57.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 202869 11.95 MB/sec warmup 93 sec latency 371.543 ms 2024-09-13T15:35:58.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 70015 4.17 MB/sec warmup 94 sec latency 131.059 ms 2024-09-13T15:35:58.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 202925 11.82 MB/sec warmup 94 sec latency 367.632 ms 2024-09-13T15:35:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:57 smithi090 ceph-mon[27811]: pgmap v319: 129 pgs: 129 active+clean; 47 MiB data, 423 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:35:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:57 smithi175 ceph-mon[36128]: pgmap v319: 129 pgs: 129 active+clean; 47 MiB data, 423 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:35:59.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 70943 4.23 MB/sec warmup 95 sec latency 12.211 ms 2024-09-13T15:35:59.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 206411 11.90 MB/sec warmup 95 sec latency 85.203 ms 2024-09-13T15:36:00.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 71511 4.20 MB/sec warmup 96 sec latency 21.134 ms 2024-09-13T15:36:00.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 208707 11.90 MB/sec warmup 96 sec latency 168.006 ms 2024-09-13T15:36:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:35:59 smithi090 ceph-mon[27811]: pgmap v320: 129 pgs: 129 active+clean; 46 MiB data, 450 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-09-13T15:36:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:35:59 smithi175 ceph-mon[36128]: pgmap v320: 129 pgs: 129 active+clean; 46 MiB data, 450 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-09-13T15:36:01.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 72347 4.21 MB/sec warmup 97 sec latency 29.022 ms 2024-09-13T15:36:01.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 211696 11.92 MB/sec warmup 97 sec latency 33.240 ms 2024-09-13T15:36:02.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 73201 4.18 MB/sec warmup 98 sec latency 12.086 ms 2024-09-13T15:36:02.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 214460 11.97 MB/sec warmup 98 sec latency 96.951 ms 2024-09-13T15:36:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:01 smithi090 ceph-mon[27811]: pgmap v321: 129 pgs: 129 active+clean; 47 MiB data, 449 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 MiB/s wr, 68 op/s 2024-09-13T15:36:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:01 smithi175 ceph-mon[36128]: pgmap v321: 129 pgs: 129 active+clean; 47 MiB data, 449 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.2 MiB/s wr, 68 op/s 2024-09-13T15:36:03.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 74274 4.22 MB/sec warmup 99 sec latency 13.523 ms 2024-09-13T15:36:03.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 217910 12.05 MB/sec warmup 99 sec latency 27.350 ms 2024-09-13T15:36:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:03 smithi090 ceph-mon[27811]: pgmap v322: 129 pgs: 129 active+clean; 48 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 MiB/s wr, 70 op/s 2024-09-13T15:36:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:04.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 74759 4.22 MB/sec warmup 100 sec latency 128.969 ms 2024-09-13T15:36:04.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 218901 11.95 MB/sec warmup 100 sec latency 370.671 ms 2024-09-13T15:36:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:03 smithi175 ceph-mon[36128]: pgmap v322: 129 pgs: 129 active+clean; 48 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 MiB/s wr, 70 op/s 2024-09-13T15:36:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:05.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 75201 4.19 MB/sec warmup 101 sec latency 68.539 ms 2024-09-13T15:36:05.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 219402 11.88 MB/sec warmup 101 sec latency 369.939 ms 2024-09-13T15:36:06.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 75935 4.19 MB/sec warmup 102 sec latency 145.144 ms 2024-09-13T15:36:06.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 220771 11.83 MB/sec warmup 102 sec latency 353.193 ms 2024-09-13T15:36:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:05 smithi090 ceph-mon[27811]: pgmap v323: 129 pgs: 129 active+clean; 51 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-13T15:36:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:05 smithi175 ceph-mon[36128]: pgmap v323: 129 pgs: 129 active+clean; 51 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 98 op/s 2024-09-13T15:36:07.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 76848 4.17 MB/sec warmup 103 sec latency 20.485 ms 2024-09-13T15:36:07.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 224255 11.90 MB/sec warmup 103 sec latency 27.806 ms 2024-09-13T15:36:08.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 77833 4.20 MB/sec warmup 104 sec latency 11.091 ms 2024-09-13T15:36:08.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 227788 11.97 MB/sec warmup 104 sec latency 41.655 ms 2024-09-13T15:36:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:07 smithi090 ceph-mon[27811]: pgmap v324: 129 pgs: 129 active+clean; 50 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-13T15:36:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:07 smithi175 ceph-mon[36128]: pgmap v324: 129 pgs: 129 active+clean; 50 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2024-09-13T15:36:09.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 78426 4.21 MB/sec warmup 105 sec latency 11.157 ms 2024-09-13T15:36:09.198 INFO:tasks.workunit.client.0.smithi090.stdout: 1 231283 12.04 MB/sec warmup 105 sec latency 33.535 ms 2024-09-13T15:36:10.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 79068 4.18 MB/sec warmup 106 sec latency 11.962 ms 2024-09-13T15:36:10.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 233800 12.05 MB/sec warmup 106 sec latency 61.546 ms 2024-09-13T15:36:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:09 smithi090 ceph-mon[27811]: pgmap v325: 129 pgs: 129 active+clean; 50 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-13T15:36:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:09 smithi175 ceph-mon[36128]: pgmap v325: 129 pgs: 129 active+clean; 50 MiB data, 472 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-13T15:36:11.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 79919 4.18 MB/sec warmup 107 sec latency 55.247 ms 2024-09-13T15:36:11.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 234887 11.99 MB/sec warmup 107 sec latency 338.523 ms 2024-09-13T15:36:12.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 80527 4.16 MB/sec warmup 108 sec latency 99.595 ms 2024-09-13T15:36:12.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 234948 11.89 MB/sec warmup 108 sec latency 371.984 ms 2024-09-13T15:36:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:11 smithi090 ceph-mon[27811]: pgmap v326: 129 pgs: 129 active+clean; 50 MiB data, 471 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:36:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:11 smithi175 ceph-mon[36128]: pgmap v326: 129 pgs: 129 active+clean; 50 MiB data, 471 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:36:13.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 81397 4.18 MB/sec warmup 109 sec latency 91.868 ms 2024-09-13T15:36:13.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 237450 11.90 MB/sec warmup 109 sec latency 355.711 ms 2024-09-13T15:36:14.004 INFO:tasks.workunit.client.1.smithi175.stdout: 1 82043 4.19 MB/sec warmup 110 sec latency 11.015 ms 2024-09-13T15:36:14.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 240615 11.95 MB/sec warmup 110 sec latency 32.126 ms 2024-09-13T15:36:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:13 smithi090 ceph-mon[27811]: pgmap v327: 129 pgs: 129 active+clean; 50 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-13T15:36:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:13 smithi175 ceph-mon[36128]: pgmap v327: 129 pgs: 129 active+clean; 50 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-13T15:36:15.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 82761 4.17 MB/sec warmup 111 sec latency 9.845 ms 2024-09-13T15:36:15.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 242231 11.91 MB/sec warmup 111 sec latency 213.374 ms 2024-09-13T15:36:16.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 83639 4.16 MB/sec warmup 112 sec latency 10.770 ms 2024-09-13T15:36:16.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 245596 11.97 MB/sec warmup 112 sec latency 49.906 ms 2024-09-13T15:36:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:15 smithi090 ceph-mon[27811]: pgmap v328: 129 pgs: 129 active+clean; 52 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 109 op/s 2024-09-13T15:36:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:36:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:36:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:15 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:36:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:15 smithi175 ceph-mon[36128]: pgmap v328: 129 pgs: 129 active+clean; 52 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 109 op/s 2024-09-13T15:36:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:36:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:36:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:15 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:36:17.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 84560 4.19 MB/sec warmup 113 sec latency 19.219 ms 2024-09-13T15:36:17.157 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:36:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:36:17.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:36:17.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:36:17.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:36:17.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:36:17.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 249013 12.03 MB/sec warmup 113 sec latency 138.285 ms 2024-09-13T15:36:18.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 85268 4.20 MB/sec warmup 114 sec latency 80.793 ms 2024-09-13T15:36:18.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 249319 11.93 MB/sec warmup 114 sec latency 325.365 ms 2024-09-13T15:36:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:17 smithi090 ceph-mon[27811]: pgmap v329: 129 pgs: 129 active+clean; 52 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:36:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:17 smithi175 ceph-mon[36128]: pgmap v329: 129 pgs: 129 active+clean; 52 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:36:19.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 85703 4.17 MB/sec warmup 115 sec latency 95.766 ms 2024-09-13T15:36:19.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 249817 11.87 MB/sec warmup 115 sec latency 425.885 ms 2024-09-13T15:36:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:20.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 86363 4.16 MB/sec warmup 116 sec latency 131.161 ms 2024-09-13T15:36:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:19 smithi175 ceph-mon[36128]: pgmap v330: 129 pgs: 129 active+clean; 54 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-09-13T15:36:20.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 251655 11.84 MB/sec warmup 116 sec latency 331.854 ms 2024-09-13T15:36:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:19 smithi090 ceph-mon[27811]: pgmap v330: 129 pgs: 129 active+clean; 54 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-09-13T15:36:21.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 87258 4.15 MB/sec warmup 117 sec latency 17.357 ms 2024-09-13T15:36:21.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 254858 11.89 MB/sec warmup 117 sec latency 37.205 ms 2024-09-13T15:36:21.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:21 smithi090 ceph-mon[27811]: pgmap v331: 129 pgs: 129 active+clean; 53 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:36:21.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:21 smithi175 ceph-mon[36128]: pgmap v331: 129 pgs: 129 active+clean; 53 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:36:22.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 88170 4.17 MB/sec warmup 118 sec latency 15.068 ms 2024-09-13T15:36:22.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 257140 11.89 MB/sec warmup 118 sec latency 146.439 ms 2024-09-13T15:36:23.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 88971 4.19 MB/sec warmup 119 sec latency 10.869 ms 2024-09-13T15:36:23.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 259896 11.91 MB/sec warmup 119 sec latency 33.434 ms 2024-09-13T15:36:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:23 smithi090 ceph-mon[27811]: pgmap v332: 129 pgs: 129 active+clean; 53 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-13T15:36:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:23 smithi175 ceph-mon[36128]: pgmap v332: 129 pgs: 129 active+clean; 53 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 84 op/s 2024-09-13T15:36:25.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 90433 4.65 MB/sec execute 1 sec latency 11.567 ms 2024-09-13T15:36:25.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 266861 19.10 MB/sec execute 1 sec latency 33.297 ms 2024-09-13T15:36:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:25 smithi090 ceph-mon[27811]: pgmap v333: 129 pgs: 129 active+clean; 54 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 108 op/s 2024-09-13T15:36:26.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 91286 3.67 MB/sec execute 2 sec latency 14.556 ms 2024-09-13T15:36:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:25 smithi175 ceph-mon[36128]: pgmap v333: 129 pgs: 129 active+clean; 54 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 108 op/s 2024-09-13T15:36:26.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 270340 19.03 MB/sec execute 2 sec latency 32.997 ms 2024-09-13T15:36:27.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 92099 4.63 MB/sec execute 3 sec latency 11.766 ms 2024-09-13T15:36:27.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 273837 19.02 MB/sec execute 3 sec latency 61.926 ms 2024-09-13T15:36:28.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:27 smithi090 ceph-mon[27811]: pgmap v334: 129 pgs: 129 active+clean; 54 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:36:28.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 92781 4.63 MB/sec execute 4 sec latency 10.436 ms 2024-09-13T15:36:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:27 smithi175 ceph-mon[36128]: pgmap v334: 129 pgs: 129 active+clean; 54 MiB data, 501 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:36:28.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 277370 19.04 MB/sec execute 4 sec latency 33.462 ms 2024-09-13T15:36:29.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 93556 4.14 MB/sec execute 5 sec latency 10.609 ms 2024-09-13T15:36:29.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 279888 17.86 MB/sec execute 5 sec latency 35.778 ms 2024-09-13T15:36:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:29 smithi090 ceph-mon[27811]: pgmap v335: 129 pgs: 129 active+clean; 57 MiB data, 522 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-13T15:36:30.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 94493 4.05 MB/sec execute 6 sec latency 9.284 ms 2024-09-13T15:36:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:29 smithi175 ceph-mon[36128]: pgmap v335: 129 pgs: 129 active+clean; 57 MiB data, 522 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-13T15:36:30.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 281882 16.72 MB/sec execute 6 sec latency 217.398 ms 2024-09-13T15:36:31.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 95550 4.70 MB/sec execute 7 sec latency 21.616 ms 2024-09-13T15:36:31.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 284711 16.43 MB/sec execute 7 sec latency 57.311 ms 2024-09-13T15:36:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:31 smithi090 ceph-mon[27811]: pgmap v336: 129 pgs: 129 active+clean; 57 MiB data, 523 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:36:32.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 96149 4.68 MB/sec execute 8 sec latency 11.940 ms 2024-09-13T15:36:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:31 smithi175 ceph-mon[36128]: pgmap v336: 129 pgs: 129 active+clean; 57 MiB data, 523 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:36:32.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 288140 16.73 MB/sec execute 8 sec latency 87.522 ms 2024-09-13T15:36:33.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 96820 4.29 MB/sec execute 9 sec latency 10.427 ms 2024-09-13T15:36:33.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 291617 16.98 MB/sec execute 9 sec latency 55.853 ms 2024-09-13T15:36:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:33 smithi090 ceph-mon[27811]: pgmap v337: 129 pgs: 129 active+clean; 57 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:36:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:34.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 97703 4.30 MB/sec execute 10 sec latency 11.583 ms 2024-09-13T15:36:34.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:33 smithi175 ceph-mon[36128]: pgmap v337: 129 pgs: 129 active+clean; 57 MiB data, 529 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:36:34.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:34.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 295156 17.19 MB/sec execute 10 sec latency 27.605 ms 2024-09-13T15:36:35.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 98579 4.42 MB/sec execute 11 sec latency 10.946 ms 2024-09-13T15:36:35.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 298652 17.35 MB/sec execute 11 sec latency 52.577 ms 2024-09-13T15:36:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:35 smithi090 ceph-mon[27811]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:36:36.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 99457 4.61 MB/sec execute 12 sec latency 11.275 ms 2024-09-13T15:36:36.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:35 smithi175 ceph-mon[36128]: pgmap v338: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:36:36.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 301277 17.09 MB/sec execute 12 sec latency 32.815 ms 2024-09-13T15:36:37.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 100027 4.38 MB/sec execute 13 sec latency 10.554 ms 2024-09-13T15:36:37.199 INFO:tasks.workunit.client.0.smithi090.stdout: 1 303416 16.56 MB/sec execute 13 sec latency 148.924 ms 2024-09-13T15:36:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:37 smithi090 ceph-mon[27811]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:36:38.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 100892 4.42 MB/sec execute 14 sec latency 11.193 ms 2024-09-13T15:36:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:37 smithi175 ceph-mon[36128]: pgmap v339: 129 pgs: 129 active+clean; 59 MiB data, 538 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:36:38.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 306705 16.72 MB/sec execute 14 sec latency 45.271 ms 2024-09-13T15:36:39.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 101753 4.25 MB/sec execute 15 sec latency 16.684 ms 2024-09-13T15:36:39.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 309499 16.58 MB/sec execute 15 sec latency 143.942 ms 2024-09-13T15:36:39.946 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:39 smithi175 ceph-mon[36128]: pgmap v340: 129 pgs: 129 active+clean; 61 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-09-13T15:36:40.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:39 smithi090 ceph-mon[27811]: pgmap v340: 129 pgs: 129 active+clean; 61 MiB data, 547 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.5 MiB/s wr, 120 op/s 2024-09-13T15:36:40.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 102674 4.45 MB/sec execute 16 sec latency 11.467 ms 2024-09-13T15:36:40.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 312968 16.73 MB/sec execute 16 sec latency 27.846 ms 2024-09-13T15:36:41.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 103287 4.46 MB/sec execute 17 sec latency 110.568 ms 2024-09-13T15:36:41.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 315450 16.56 MB/sec execute 17 sec latency 397.326 ms 2024-09-13T15:36:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:41 smithi090 ceph-mon[27811]: pgmap v341: 129 pgs: 129 active+clean; 60 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:36:42.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 103928 4.29 MB/sec execute 18 sec latency 13.431 ms 2024-09-13T15:36:42.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:41 smithi175 ceph-mon[36128]: pgmap v341: 129 pgs: 129 active+clean; 60 MiB data, 550 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:36:42.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 318470 16.52 MB/sec execute 18 sec latency 40.901 ms 2024-09-13T15:36:43.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 104797 4.28 MB/sec execute 19 sec latency 11.128 ms 2024-09-13T15:36:43.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 321624 16.44 MB/sec execute 19 sec latency 33.483 ms 2024-09-13T15:36:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:43 smithi090 ceph-mon[27811]: pgmap v342: 129 pgs: 129 active+clean; 61 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 89 op/s 2024-09-13T15:36:44.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 105668 4.34 MB/sec execute 20 sec latency 14.148 ms 2024-09-13T15:36:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:43 smithi175 ceph-mon[36128]: pgmap v342: 129 pgs: 129 active+clean; 61 MiB data, 553 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 89 op/s 2024-09-13T15:36:44.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 324433 16.49 MB/sec execute 20 sec latency 63.963 ms 2024-09-13T15:36:45.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 106539 4.46 MB/sec execute 21 sec latency 12.298 ms 2024-09-13T15:36:45.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 327327 16.41 MB/sec execute 21 sec latency 32.262 ms 2024-09-13T15:36:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:45 smithi090 ceph-mon[27811]: pgmap v343: 129 pgs: 129 active+clean; 62 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.2 MiB/s wr, 116 op/s 2024-09-13T15:36:46.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 107107 4.32 MB/sec execute 22 sec latency 13.433 ms 2024-09-13T15:36:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:45 smithi175 ceph-mon[36128]: pgmap v343: 129 pgs: 129 active+clean; 62 MiB data, 558 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.2 MiB/s wr, 116 op/s 2024-09-13T15:36:46.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330571 16.50 MB/sec execute 22 sec latency 272.169 ms 2024-09-13T15:36:47.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 107629 4.22 MB/sec execute 23 sec latency 140.783 ms 2024-09-13T15:36:47.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330620 15.79 MB/sec execute 23 sec latency 359.054 ms 2024-09-13T15:36:48.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:47 smithi090 ceph-mon[27811]: pgmap v344: 129 pgs: 129 active+clean; 63 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:36:48.005 INFO:tasks.workunit.client.1.smithi175.stdout: 1 108325 4.18 MB/sec execute 24 sec latency 134.598 ms 2024-09-13T15:36:48.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:47 smithi175 ceph-mon[36128]: pgmap v344: 129 pgs: 129 active+clean; 63 MiB data, 559 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:36:48.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330627 15.13 MB/sec execute 24 sec latency 369.515 ms 2024-09-13T15:36:49.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:49.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 108883 4.09 MB/sec execute 25 sec latency 140.125 ms 2024-09-13T15:36:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:36:49.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330688 14.53 MB/sec execute 25 sec latency 372.387 ms 2024-09-13T15:36:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:49 smithi090 ceph-mon[27811]: pgmap v345: 129 pgs: 129 active+clean; 63 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.3 MiB/s wr, 112 op/s 2024-09-13T15:36:50.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 109736 4.21 MB/sec execute 26 sec latency 111.066 ms 2024-09-13T15:36:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:49 smithi175 ceph-mon[36128]: pgmap v345: 129 pgs: 129 active+clean; 63 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.3 MiB/s wr, 112 op/s 2024-09-13T15:36:50.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330695 13.97 MB/sec execute 26 sec latency 347.716 ms 2024-09-13T15:36:51.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 110220 4.22 MB/sec execute 27 sec latency 114.713 ms 2024-09-13T15:36:51.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330769 13.46 MB/sec execute 27 sec latency 341.292 ms 2024-09-13T15:36:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:51 smithi090 ceph-mon[27811]: pgmap v346: 129 pgs: 129 active+clean; 63 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-13T15:36:52.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 110594 4.08 MB/sec execute 28 sec latency 137.758 ms 2024-09-13T15:36:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:51 smithi175 ceph-mon[36128]: pgmap v346: 129 pgs: 129 active+clean; 63 MiB data, 572 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.5 MiB/s wr, 79 op/s 2024-09-13T15:36:52.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 330812 12.98 MB/sec execute 28 sec latency 380.956 ms 2024-09-13T15:36:53.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 111164 4.01 MB/sec execute 29 sec latency 129.150 ms 2024-09-13T15:36:53.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 331373 12.68 MB/sec execute 29 sec latency 397.805 ms 2024-09-13T15:36:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:53 smithi090 ceph-mon[27811]: pgmap v347: 129 pgs: 129 active+clean; 62 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:36:54.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 111704 3.98 MB/sec execute 30 sec latency 143.914 ms 2024-09-13T15:36:54.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:53 smithi175 ceph-mon[36128]: pgmap v347: 129 pgs: 129 active+clean; 62 MiB data, 560 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:36:54.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 331775 12.27 MB/sec execute 30 sec latency 385.604 ms 2024-09-13T15:36:55.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 112341 3.90 MB/sec execute 31 sec latency 145.351 ms 2024-09-13T15:36:55.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 331779 11.88 MB/sec execute 31 sec latency 364.505 ms 2024-09-13T15:36:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:55 smithi090 ceph-mon[27811]: pgmap v348: 129 pgs: 129 active+clean; 64 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-09-13T15:36:56.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 113020 3.96 MB/sec execute 32 sec latency 147.945 ms 2024-09-13T15:36:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:55 smithi175 ceph-mon[36128]: pgmap v348: 129 pgs: 129 active+clean; 64 MiB data, 571 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-09-13T15:36:56.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 332303 11.56 MB/sec execute 32 sec latency 362.228 ms 2024-09-13T15:36:57.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 113769 4.03 MB/sec execute 33 sec latency 98.094 ms 2024-09-13T15:36:57.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 334421 11.60 MB/sec execute 33 sec latency 341.352 ms 2024-09-13T15:36:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:57 smithi090 ceph-mon[27811]: pgmap v349: 129 pgs: 129 active+clean; 63 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 72 op/s 2024-09-13T15:36:58.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 114366 3.93 MB/sec execute 34 sec latency 11.065 ms 2024-09-13T15:36:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:57 smithi175 ceph-mon[36128]: pgmap v349: 129 pgs: 129 active+clean; 63 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 72 op/s 2024-09-13T15:36:58.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 337860 11.81 MB/sec execute 34 sec latency 41.780 ms 2024-09-13T15:36:59.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 115187 3.96 MB/sec execute 35 sec latency 11.362 ms 2024-09-13T15:36:59.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 341326 12.02 MB/sec execute 35 sec latency 33.335 ms 2024-09-13T15:37:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:36:59 smithi090 ceph-mon[27811]: pgmap v350: 129 pgs: 129 active+clean; 63 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:37:00.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 116012 3.91 MB/sec execute 36 sec latency 17.849 ms 2024-09-13T15:37:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:36:59 smithi175 ceph-mon[36128]: pgmap v350: 129 pgs: 129 active+clean; 63 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:37:00.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 344812 12.21 MB/sec execute 36 sec latency 33.534 ms 2024-09-13T15:37:01.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 116873 3.99 MB/sec execute 37 sec latency 11.429 ms 2024-09-13T15:37:01.200 INFO:tasks.workunit.client.0.smithi090.stdout: 1 348296 12.39 MB/sec execute 37 sec latency 63.275 ms 2024-09-13T15:37:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:01 smithi090 ceph-mon[27811]: pgmap v351: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 MiB/s wr, 71 op/s 2024-09-13T15:37:02.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 117475 4.01 MB/sec execute 38 sec latency 90.298 ms 2024-09-13T15:37:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:01 smithi175 ceph-mon[36128]: pgmap v351: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.3 MiB/s wr, 71 op/s 2024-09-13T15:37:02.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 348348 12.07 MB/sec execute 38 sec latency 339.432 ms 2024-09-13T15:37:03.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 117994 3.93 MB/sec execute 39 sec latency 94.865 ms 2024-09-13T15:37:03.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 348355 11.76 MB/sec execute 39 sec latency 375.836 ms 2024-09-13T15:37:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:03 smithi090 ceph-mon[27811]: pgmap v352: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-13T15:37:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:04.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 118737 3.94 MB/sec execute 40 sec latency 128.724 ms 2024-09-13T15:37:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:03 smithi175 ceph-mon[36128]: pgmap v352: 129 pgs: 129 active+clean; 65 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-13T15:37:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:04.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 350912 11.82 MB/sec execute 40 sec latency 268.109 ms 2024-09-13T15:37:05.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 119576 3.91 MB/sec execute 41 sec latency 17.565 ms 2024-09-13T15:37:05.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 353536 11.82 MB/sec execute 41 sec latency 136.271 ms 2024-09-13T15:37:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:05 smithi090 ceph-mon[27811]: pgmap v353: 129 pgs: 129 active+clean; 67 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.8 MiB/s wr, 99 op/s 2024-09-13T15:37:06.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 120425 3.97 MB/sec execute 42 sec latency 11.352 ms 2024-09-13T15:37:06.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:05 smithi175 ceph-mon[36128]: pgmap v353: 129 pgs: 129 active+clean; 67 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.8 MiB/s wr, 99 op/s 2024-09-13T15:37:06.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 356592 11.95 MB/sec execute 42 sec latency 32.140 ms 2024-09-13T15:37:07.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 121112 3.99 MB/sec execute 43 sec latency 10.954 ms 2024-09-13T15:37:07.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 359737 12.11 MB/sec execute 43 sec latency 49.204 ms 2024-09-13T15:37:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:07 smithi090 ceph-mon[27811]: pgmap v354: 129 pgs: 129 active+clean; 67 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.4 MiB/s wr, 75 op/s 2024-09-13T15:37:08.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 121886 3.95 MB/sec execute 44 sec latency 10.260 ms 2024-09-13T15:37:08.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:07 smithi175 ceph-mon[36128]: pgmap v354: 129 pgs: 129 active+clean; 67 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.4 MiB/s wr, 75 op/s 2024-09-13T15:37:08.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 362601 12.17 MB/sec execute 44 sec latency 139.971 ms 2024-09-13T15:37:09.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 122715 3.94 MB/sec execute 45 sec latency 60.334 ms 2024-09-13T15:37:09.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 363429 12.00 MB/sec execute 45 sec latency 341.299 ms 2024-09-13T15:37:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:09 smithi090 ceph-mon[27811]: pgmap v355: 129 pgs: 129 active+clean; 68 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-13T15:37:10.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 123250 3.90 MB/sec execute 46 sec latency 107.932 ms 2024-09-13T15:37:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:09 smithi175 ceph-mon[36128]: pgmap v355: 129 pgs: 129 active+clean; 68 MiB data, 597 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-13T15:37:10.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 363683 11.74 MB/sec execute 46 sec latency 388.325 ms 2024-09-13T15:37:11.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 124021 3.96 MB/sec execute 47 sec latency 103.694 ms 2024-09-13T15:37:11.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 366084 11.79 MB/sec execute 47 sec latency 360.288 ms 2024-09-13T15:37:11.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:11 smithi090 ceph-mon[27811]: pgmap v356: 129 pgs: 129 active+clean; 68 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-13T15:37:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:11 smithi175 ceph-mon[36128]: pgmap v356: 129 pgs: 129 active+clean; 68 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 94 op/s 2024-09-13T15:37:12.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 124706 3.97 MB/sec execute 48 sec latency 14.502 ms 2024-09-13T15:37:12.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 369562 11.94 MB/sec execute 48 sec latency 112.847 ms 2024-09-13T15:37:13.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 125469 3.93 MB/sec execute 49 sec latency 11.668 ms 2024-09-13T15:37:13.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 372210 11.99 MB/sec execute 49 sec latency 33.314 ms 2024-09-13T15:37:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:13 smithi090 ceph-mon[27811]: pgmap v357: 129 pgs: 129 active+clean; 68 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-13T15:37:14.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 126380 3.93 MB/sec execute 50 sec latency 14.090 ms 2024-09-13T15:37:14.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:13 smithi175 ceph-mon[36128]: pgmap v357: 129 pgs: 129 active+clean; 68 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 78 op/s 2024-09-13T15:37:14.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 375411 12.10 MB/sec execute 50 sec latency 33.481 ms 2024-09-13T15:37:15.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 127463 4.02 MB/sec execute 51 sec latency 10.816 ms 2024-09-13T15:37:15.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 378360 12.12 MB/sec execute 51 sec latency 58.495 ms 2024-09-13T15:37:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:15 smithi090 ceph-mon[27811]: pgmap v358: 129 pgs: 129 active+clean; 70 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 104 op/s 2024-09-13T15:37:16.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 128038 4.03 MB/sec execute 52 sec latency 11.759 ms 2024-09-13T15:37:16.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:15 smithi175 ceph-mon[36128]: pgmap v358: 129 pgs: 129 active+clean; 70 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 104 op/s 2024-09-13T15:37:16.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 381454 12.23 MB/sec execute 52 sec latency 33.114 ms 2024-09-13T15:37:17.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:37:17.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:37:17.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:37:17.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:37:17.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 128635 3.97 MB/sec execute 53 sec latency 11.775 ms 2024-09-13T15:37:17.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:37:17.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:37:17.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:37:17.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:37:17.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 384097 12.27 MB/sec execute 53 sec latency 33.263 ms 2024-09-13T15:37:18.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:17 smithi090 ceph-mon[27811]: pgmap v359: 129 pgs: 129 active+clean; 70 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-13T15:37:18.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 129469 3.98 MB/sec execute 54 sec latency 11.464 ms 2024-09-13T15:37:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:17 smithi175 ceph-mon[36128]: pgmap v359: 129 pgs: 129 active+clean; 70 MiB data, 626 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 76 op/s 2024-09-13T15:37:18.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 387489 12.39 MB/sec execute 54 sec latency 46.588 ms 2024-09-13T15:37:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:19.006 INFO:tasks.workunit.client.1.smithi175.stdout: 1 130298 3.96 MB/sec execute 55 sec latency 18.482 ms 2024-09-13T15:37:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:19.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 390959 12.51 MB/sec execute 55 sec latency 109.281 ms 2024-09-13T15:37:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:19 smithi090 ceph-mon[27811]: pgmap v360: 129 pgs: 129 active+clean; 71 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:37:20.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 131221 4.06 MB/sec execute 56 sec latency 11.951 ms 2024-09-13T15:37:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:19 smithi175 ceph-mon[36128]: pgmap v360: 129 pgs: 129 active+clean; 71 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:37:20.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 394455 12.63 MB/sec execute 56 sec latency 33.208 ms 2024-09-13T15:37:21.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 131819 4.02 MB/sec execute 57 sec latency 11.646 ms 2024-09-13T15:37:21.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 397431 12.72 MB/sec execute 57 sec latency 42.806 ms 2024-09-13T15:37:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:21 smithi090 ceph-mon[27811]: pgmap v361: 129 pgs: 129 active+clean; 71 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 93 op/s 2024-09-13T15:37:22.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 132567 3.98 MB/sec execute 58 sec latency 14.152 ms 2024-09-13T15:37:22.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:21 smithi175 ceph-mon[36128]: pgmap v361: 129 pgs: 129 active+clean; 71 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 93 op/s 2024-09-13T15:37:22.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 400562 12.76 MB/sec execute 58 sec latency 39.376 ms 2024-09-13T15:37:23.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 133494 3.98 MB/sec execute 59 sec latency 17.246 ms 2024-09-13T15:37:23.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 403624 12.82 MB/sec execute 59 sec latency 33.292 ms 2024-09-13T15:37:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:23 smithi090 ceph-mon[27811]: pgmap v362: 129 pgs: 129 active+clean; 72 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:37:24.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 134558 4.05 MB/sec execute 60 sec latency 9.312 ms 2024-09-13T15:37:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:23 smithi175 ceph-mon[36128]: pgmap v362: 129 pgs: 129 active+clean; 72 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:37:24.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 406292 12.83 MB/sec execute 60 sec latency 101.796 ms 2024-09-13T15:37:25.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 135158 4.06 MB/sec execute 61 sec latency 10.857 ms 2024-09-13T15:37:25.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 408941 12.86 MB/sec execute 61 sec latency 41.617 ms 2024-09-13T15:37:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:25 smithi090 ceph-mon[27811]: pgmap v363: 129 pgs: 129 active+clean; 72 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.0 MiB/s wr, 112 op/s 2024-09-13T15:37:26.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 135813 4.02 MB/sec execute 62 sec latency 11.126 ms 2024-09-13T15:37:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:25 smithi175 ceph-mon[36128]: pgmap v363: 129 pgs: 129 active+clean; 72 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.0 MiB/s wr, 112 op/s 2024-09-13T15:37:26.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 412375 12.96 MB/sec execute 62 sec latency 34.511 ms 2024-09-13T15:37:27.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 136693 4.02 MB/sec execute 63 sec latency 10.466 ms 2024-09-13T15:37:27.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 415843 13.05 MB/sec execute 63 sec latency 33.284 ms 2024-09-13T15:37:28.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:27 smithi090 ceph-mon[27811]: pgmap v364: 129 pgs: 129 active+clean; 73 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:37:28.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 137578 4.05 MB/sec execute 64 sec latency 14.628 ms 2024-09-13T15:37:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:27 smithi175 ceph-mon[36128]: pgmap v364: 129 pgs: 129 active+clean; 73 MiB data, 637 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:37:28.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 419321 13.14 MB/sec execute 64 sec latency 44.945 ms 2024-09-13T15:37:29.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 138438 4.09 MB/sec execute 65 sec latency 11.823 ms 2024-09-13T15:37:29.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 422815 13.23 MB/sec execute 65 sec latency 55.957 ms 2024-09-13T15:37:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:29 smithi090 ceph-mon[27811]: pgmap v365: 129 pgs: 129 active+clean; 72 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-09-13T15:37:30.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 139025 4.05 MB/sec execute 66 sec latency 9.341 ms 2024-09-13T15:37:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:29 smithi175 ceph-mon[36128]: pgmap v365: 129 pgs: 129 active+clean; 72 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-09-13T15:37:30.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 425467 13.25 MB/sec execute 66 sec latency 33.698 ms 2024-09-13T15:37:31.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 139873 4.06 MB/sec execute 67 sec latency 10.764 ms 2024-09-13T15:37:31.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 427563 13.20 MB/sec execute 67 sec latency 298.544 ms 2024-09-13T15:37:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:31 smithi090 ceph-mon[27811]: pgmap v366: 129 pgs: 129 active+clean; 72 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-13T15:37:32.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 140676 4.03 MB/sec execute 68 sec latency 111.290 ms 2024-09-13T15:37:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:31 smithi175 ceph-mon[36128]: pgmap v366: 129 pgs: 129 active+clean; 72 MiB data, 621 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-13T15:37:32.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 430032 13.22 MB/sec execute 68 sec latency 389.170 ms 2024-09-13T15:37:33.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 141651 4.08 MB/sec execute 69 sec latency 11.355 ms 2024-09-13T15:37:33.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 433454 13.30 MB/sec execute 69 sec latency 146.554 ms 2024-09-13T15:37:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:33 smithi090 ceph-mon[27811]: pgmap v367: 129 pgs: 129 active+clean; 72 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-13T15:37:34.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:34.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 142234 4.08 MB/sec execute 70 sec latency 11.088 ms 2024-09-13T15:37:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:33 smithi175 ceph-mon[36128]: pgmap v367: 129 pgs: 129 active+clean; 72 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.8 MiB/s wr, 89 op/s 2024-09-13T15:37:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:34.201 INFO:tasks.workunit.client.0.smithi090.stdout: 1 435762 13.28 MB/sec execute 70 sec latency 133.632 ms 2024-09-13T15:37:35.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 142890 4.04 MB/sec execute 71 sec latency 10.795 ms 2024-09-13T15:37:35.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 438691 13.29 MB/sec execute 71 sec latency 50.209 ms 2024-09-13T15:37:35.945 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:35 smithi175 ceph-mon[36128]: pgmap v368: 129 pgs: 129 active+clean; 75 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:37:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:35 smithi090 ceph-mon[27811]: pgmap v368: 129 pgs: 129 active+clean; 75 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:37:36.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 143772 4.05 MB/sec execute 72 sec latency 10.813 ms 2024-09-13T15:37:36.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 440848 13.28 MB/sec execute 72 sec latency 140.522 ms 2024-09-13T15:37:37.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 144484 4.03 MB/sec execute 73 sec latency 127.707 ms 2024-09-13T15:37:37.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 443050 13.27 MB/sec execute 73 sec latency 373.749 ms 2024-09-13T15:37:38.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:37 smithi090 ceph-mon[27811]: pgmap v369: 129 pgs: 129 active+clean; 74 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-13T15:37:38.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 145218 4.06 MB/sec execute 74 sec latency 139.159 ms 2024-09-13T15:37:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:37 smithi175 ceph-mon[36128]: pgmap v369: 129 pgs: 129 active+clean; 74 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-13T15:37:38.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 443182 13.11 MB/sec execute 74 sec latency 344.489 ms 2024-09-13T15:37:39.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 145774 4.07 MB/sec execute 75 sec latency 124.124 ms 2024-09-13T15:37:39.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444083 13.00 MB/sec execute 75 sec latency 376.332 ms 2024-09-13T15:37:40.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:39 smithi090 ceph-mon[27811]: pgmap v370: 129 pgs: 129 active+clean; 75 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:37:40.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 146154 4.02 MB/sec execute 76 sec latency 131.444 ms 2024-09-13T15:37:40.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:39 smithi175 ceph-mon[36128]: pgmap v370: 129 pgs: 129 active+clean; 75 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:37:40.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444090 12.83 MB/sec execute 76 sec latency 359.856 ms 2024-09-13T15:37:41.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 146862 4.03 MB/sec execute 77 sec latency 101.878 ms 2024-09-13T15:37:41.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444137 12.66 MB/sec execute 77 sec latency 379.922 ms 2024-09-13T15:37:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:41 smithi090 ceph-mon[27811]: pgmap v371: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:37:42.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 147665 3.99 MB/sec execute 78 sec latency 44.990 ms 2024-09-13T15:37:42.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:41 smithi175 ceph-mon[36128]: pgmap v371: 129 pgs: 129 active+clean; 74 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:37:42.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444144 12.50 MB/sec execute 78 sec latency 370.709 ms 2024-09-13T15:37:43.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 148375 4.02 MB/sec execute 79 sec latency 97.331 ms 2024-09-13T15:37:43.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444205 12.34 MB/sec execute 79 sec latency 392.924 ms 2024-09-13T15:37:44.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:43 smithi090 ceph-mon[27811]: pgmap v372: 129 pgs: 129 active+clean; 76 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-13T15:37:44.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 148793 4.00 MB/sec execute 80 sec latency 137.446 ms 2024-09-13T15:37:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:43 smithi175 ceph-mon[36128]: pgmap v372: 129 pgs: 129 active+clean; 76 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-13T15:37:44.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444212 12.19 MB/sec execute 80 sec latency 368.278 ms 2024-09-13T15:37:45.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 149321 4.00 MB/sec execute 81 sec latency 142.874 ms 2024-09-13T15:37:45.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444217 12.04 MB/sec execute 81 sec latency 351.496 ms 2024-09-13T15:37:46.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:45 smithi090 ceph-mon[27811]: pgmap v373: 129 pgs: 129 active+clean; 78 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-09-13T15:37:46.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 149884 3.97 MB/sec execute 82 sec latency 124.469 ms 2024-09-13T15:37:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:45 smithi175 ceph-mon[36128]: pgmap v373: 129 pgs: 129 active+clean; 78 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-09-13T15:37:46.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 444328 11.90 MB/sec execute 82 sec latency 364.542 ms 2024-09-13T15:37:47.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 150650 3.97 MB/sec execute 83 sec latency 72.934 ms 2024-09-13T15:37:47.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 446599 11.90 MB/sec execute 83 sec latency 393.843 ms 2024-09-13T15:37:48.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:47 smithi090 ceph-mon[27811]: pgmap v374: 129 pgs: 129 active+clean; 78 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-09-13T15:37:48.007 INFO:tasks.workunit.client.1.smithi175.stdout: 1 151494 3.96 MB/sec execute 84 sec latency 10.309 ms 2024-09-13T15:37:48.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:47 smithi175 ceph-mon[36128]: pgmap v374: 129 pgs: 129 active+clean; 78 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-09-13T15:37:48.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 449425 11.92 MB/sec execute 84 sec latency 39.406 ms 2024-09-13T15:37:49.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:49.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 152335 3.99 MB/sec execute 85 sec latency 11.466 ms 2024-09-13T15:37:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:37:49.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 452389 11.98 MB/sec execute 85 sec latency 33.133 ms 2024-09-13T15:37:50.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:49 smithi090 ceph-mon[27811]: pgmap v375: 129 pgs: 129 active+clean; 78 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-13T15:37:50.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 153019 4.00 MB/sec execute 86 sec latency 12.311 ms 2024-09-13T15:37:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:49 smithi175 ceph-mon[36128]: pgmap v375: 129 pgs: 129 active+clean; 78 MiB data, 672 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-13T15:37:50.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 454961 12.01 MB/sec execute 86 sec latency 98.813 ms 2024-09-13T15:37:51.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 153755 3.97 MB/sec execute 87 sec latency 11.173 ms 2024-09-13T15:37:51.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 459072 12.33 MB/sec execute 87 sec latency 33.363 ms 2024-09-13T15:37:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:51 smithi090 ceph-mon[27811]: pgmap v376: 129 pgs: 129 active+clean; 79 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 MiB/s wr, 69 op/s 2024-09-13T15:37:52.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 154614 3.97 MB/sec execute 88 sec latency 41.736 ms 2024-09-13T15:37:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:51 smithi175 ceph-mon[36128]: pgmap v376: 129 pgs: 129 active+clean; 79 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 MiB/s wr, 69 op/s 2024-09-13T15:37:52.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 461326 12.33 MB/sec execute 88 sec latency 304.875 ms 2024-09-13T15:37:53.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 155111 3.95 MB/sec execute 89 sec latency 143.093 ms 2024-09-13T15:37:53.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 461641 12.21 MB/sec execute 89 sec latency 362.107 ms 2024-09-13T15:37:54.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:53 smithi090 ceph-mon[27811]: pgmap v377: 129 pgs: 129 active+clean; 79 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-09-13T15:37:54.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 155871 3.98 MB/sec execute 90 sec latency 124.545 ms 2024-09-13T15:37:54.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:53 smithi175 ceph-mon[36128]: pgmap v377: 129 pgs: 129 active+clean; 79 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-09-13T15:37:54.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 462587 12.13 MB/sec execute 90 sec latency 380.268 ms 2024-09-13T15:37:55.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 156557 3.99 MB/sec execute 91 sec latency 10.105 ms 2024-09-13T15:37:55.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 465979 12.20 MB/sec execute 91 sec latency 33.433 ms 2024-09-13T15:37:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:55 smithi090 ceph-mon[27811]: pgmap v378: 129 pgs: 129 active+clean; 79 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 101 op/s 2024-09-13T15:37:56.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 157314 3.97 MB/sec execute 92 sec latency 10.184 ms 2024-09-13T15:37:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:55 smithi175 ceph-mon[36128]: pgmap v378: 129 pgs: 129 active+clean; 79 MiB data, 690 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.8 MiB/s wr, 101 op/s 2024-09-13T15:37:56.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 468640 12.22 MB/sec execute 92 sec latency 34.259 ms 2024-09-13T15:37:57.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 158236 3.96 MB/sec execute 93 sec latency 12.554 ms 2024-09-13T15:37:57.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 471763 12.27 MB/sec execute 93 sec latency 70.352 ms 2024-09-13T15:37:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:57 smithi090 ceph-mon[27811]: pgmap v379: 129 pgs: 129 active+clean; 80 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 73 op/s 2024-09-13T15:37:58.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 159121 3.99 MB/sec execute 94 sec latency 12.320 ms 2024-09-13T15:37:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:57 smithi175 ceph-mon[36128]: pgmap v379: 129 pgs: 129 active+clean; 80 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 73 op/s 2024-09-13T15:37:58.202 INFO:tasks.workunit.client.0.smithi090.stdout: 1 474433 12.28 MB/sec execute 94 sec latency 44.474 ms 2024-09-13T15:37:59.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 159912 4.02 MB/sec execute 95 sec latency 10.572 ms 2024-09-13T15:37:59.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 477011 12.30 MB/sec execute 95 sec latency 142.646 ms 2024-09-13T15:38:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:37:59 smithi090 ceph-mon[27811]: pgmap v380: 129 pgs: 129 active+clean; 82 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:38:00.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 160325 3.98 MB/sec execute 96 sec latency 103.611 ms 2024-09-13T15:38:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:37:59 smithi175 ceph-mon[36128]: pgmap v380: 129 pgs: 129 active+clean; 82 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:38:00.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 477080 12.18 MB/sec execute 96 sec latency 377.346 ms 2024-09-13T15:38:01.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 161042 3.99 MB/sec execute 97 sec latency 136.906 ms 2024-09-13T15:38:01.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 477979 12.10 MB/sec execute 97 sec latency 373.290 ms 2024-09-13T15:38:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:01 smithi090 ceph-mon[27811]: pgmap v381: 129 pgs: 129 active+clean; 82 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:38:02.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 161889 3.96 MB/sec execute 98 sec latency 18.011 ms 2024-09-13T15:38:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:01 smithi175 ceph-mon[36128]: pgmap v381: 129 pgs: 129 active+clean; 82 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:38:02.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 480592 12.12 MB/sec execute 98 sec latency 80.051 ms 2024-09-13T15:38:03.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 162924 4.00 MB/sec execute 99 sec latency 9.911 ms 2024-09-13T15:38:03.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 484021 12.19 MB/sec execute 99 sec latency 122.443 ms 2024-09-13T15:38:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:03 smithi090 ceph-mon[27811]: pgmap v382: 129 pgs: 129 active+clean; 82 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:38:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:04.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 163526 4.01 MB/sec execute 100 sec latency 10.035 ms 2024-09-13T15:38:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:03 smithi175 ceph-mon[36128]: pgmap v382: 129 pgs: 129 active+clean; 82 MiB data, 689 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:38:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:04.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 487500 12.26 MB/sec execute 100 sec latency 36.375 ms 2024-09-13T15:38:05.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 164187 3.98 MB/sec execute 101 sec latency 10.917 ms 2024-09-13T15:38:05.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 490994 12.32 MB/sec execute 101 sec latency 33.243 ms 2024-09-13T15:38:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:05 smithi090 ceph-mon[27811]: pgmap v383: 129 pgs: 129 active+clean; 82 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 106 op/s 2024-09-13T15:38:06.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 165073 3.98 MB/sec execute 102 sec latency 10.470 ms 2024-09-13T15:38:06.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:05 smithi175 ceph-mon[36128]: pgmap v383: 129 pgs: 129 active+clean; 82 MiB data, 687 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 106 op/s 2024-09-13T15:38:06.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 494528 12.39 MB/sec execute 102 sec latency 33.617 ms 2024-09-13T15:38:07.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 165684 3.97 MB/sec execute 103 sec latency 126.597 ms 2024-09-13T15:38:07.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 494537 12.27 MB/sec execute 103 sec latency 351.285 ms 2024-09-13T15:38:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:07 smithi090 ceph-mon[27811]: pgmap v384: 129 pgs: 129 active+clean; 82 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:38:08.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 166474 3.99 MB/sec execute 104 sec latency 127.987 ms 2024-09-13T15:38:08.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:07 smithi175 ceph-mon[36128]: pgmap v384: 129 pgs: 129 active+clean; 82 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:38:08.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 494589 12.15 MB/sec execute 104 sec latency 356.898 ms 2024-09-13T15:38:09.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 167098 4.00 MB/sec execute 105 sec latency 11.735 ms 2024-09-13T15:38:09.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 498081 12.22 MB/sec execute 105 sec latency 63.480 ms 2024-09-13T15:38:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:09 smithi090 ceph-mon[27811]: pgmap v385: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-13T15:38:10.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 167769 3.98 MB/sec execute 106 sec latency 12.514 ms 2024-09-13T15:38:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:09 smithi175 ceph-mon[36128]: pgmap v385: 129 pgs: 129 active+clean; 84 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-13T15:38:10.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 499880 12.17 MB/sec execute 106 sec latency 159.232 ms 2024-09-13T15:38:11.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 168637 3.98 MB/sec execute 107 sec latency 13.110 ms 2024-09-13T15:38:11.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 502833 12.22 MB/sec execute 107 sec latency 33.435 ms 2024-09-13T15:38:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:11 smithi090 ceph-mon[27811]: pgmap v386: 129 pgs: 129 active+clean; 84 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:38:12.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 169494 3.99 MB/sec execute 108 sec latency 13.140 ms 2024-09-13T15:38:12.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:11 smithi175 ceph-mon[36128]: pgmap v386: 129 pgs: 129 active+clean; 84 MiB data, 682 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:38:12.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 505559 12.25 MB/sec execute 108 sec latency 40.493 ms 2024-09-13T15:38:13.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 170382 4.02 MB/sec execute 109 sec latency 10.852 ms 2024-09-13T15:38:13.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 508919 12.30 MB/sec execute 109 sec latency 32.901 ms 2024-09-13T15:38:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:13 smithi090 ceph-mon[27811]: pgmap v387: 129 pgs: 129 active+clean; 85 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:38:14.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 170965 3.99 MB/sec execute 110 sec latency 9.121 ms 2024-09-13T15:38:14.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:13 smithi175 ceph-mon[36128]: pgmap v387: 129 pgs: 129 active+clean; 85 MiB data, 688 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:38:14.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 512388 12.36 MB/sec execute 110 sec latency 57.430 ms 2024-09-13T15:38:15.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 171770 4.00 MB/sec execute 111 sec latency 14.858 ms 2024-09-13T15:38:15.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 515869 12.42 MB/sec execute 111 sec latency 26.397 ms 2024-09-13T15:38:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:15 smithi090 ceph-mon[27811]: pgmap v388: 129 pgs: 129 active+clean; 85 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:38:16.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 172646 3.98 MB/sec execute 112 sec latency 10.650 ms 2024-09-13T15:38:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:15 smithi175 ceph-mon[36128]: pgmap v388: 129 pgs: 129 active+clean; 85 MiB data, 692 MiB used, 715 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:38:16.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 519405 12.48 MB/sec execute 112 sec latency 34.288 ms 2024-09-13T15:38:17.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:16 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:38:17.008 INFO:tasks.workunit.client.1.smithi175.stdout: 1 173558 4.01 MB/sec execute 113 sec latency 108.272 ms 2024-09-13T15:38:17.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:16 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:38:17.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 522892 12.54 MB/sec execute 113 sec latency 41.744 ms 2024-09-13T15:38:18.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 174133 4.02 MB/sec execute 114 sec latency 27.917 ms 2024-09-13T15:38:18.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 525517 12.55 MB/sec execute 114 sec latency 33.310 ms 2024-09-13T15:38:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:17 smithi090 ceph-mon[27811]: pgmap v389: 129 pgs: 129 active+clean; 86 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:38:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:38:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:38:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:38:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:38:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:38:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:17 smithi175 ceph-mon[36128]: pgmap v389: 129 pgs: 129 active+clean; 86 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:38:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:38:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:38:18.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:38:18.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:38:18.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:38:19.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 174769 3.99 MB/sec execute 115 sec latency 10.997 ms 2024-09-13T15:38:19.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:19.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 528743 12.59 MB/sec execute 115 sec latency 35.836 ms 2024-09-13T15:38:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:20.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 175650 4.00 MB/sec execute 116 sec latency 14.053 ms 2024-09-13T15:38:20.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 530947 12.59 MB/sec execute 116 sec latency 99.158 ms 2024-09-13T15:38:20.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:19 smithi090 ceph-mon[27811]: pgmap v390: 129 pgs: 129 active+clean; 86 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.3 MiB/s wr, 109 op/s 2024-09-13T15:38:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:19 smithi175 ceph-mon[36128]: pgmap v390: 129 pgs: 129 active+clean; 86 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.3 MiB/s wr, 109 op/s 2024-09-13T15:38:21.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 176413 3.99 MB/sec execute 117 sec latency 19.835 ms 2024-09-13T15:38:21.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 533814 12.61 MB/sec execute 117 sec latency 83.789 ms 2024-09-13T15:38:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:21 smithi090 ceph-mon[27811]: pgmap v391: 129 pgs: 129 active+clean; 86 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:38:22.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 177332 4.03 MB/sec execute 118 sec latency 12.771 ms 2024-09-13T15:38:22.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:21 smithi175 ceph-mon[36128]: pgmap v391: 129 pgs: 129 active+clean; 86 MiB data, 707 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:38:22.203 INFO:tasks.workunit.client.0.smithi090.stdout: 1 537238 12.66 MB/sec execute 118 sec latency 41.389 ms 2024-09-13T15:38:23.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 177922 4.01 MB/sec execute 119 sec latency 11.149 ms 2024-09-13T15:38:23.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 540724 12.71 MB/sec execute 119 sec latency 32.885 ms 2024-09-13T15:38:24.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 178753 4.02 MB/sec execute 120 sec latency 12.270 ms 2024-09-13T15:38:24.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 544255 12.76 MB/sec execute 120 sec latency 31.434 ms 2024-09-13T15:38:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:23 smithi090 ceph-mon[27811]: pgmap v392: 129 pgs: 129 active+clean; 86 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-13T15:38:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:23 smithi175 ceph-mon[36128]: pgmap v392: 129 pgs: 129 active+clean; 86 MiB data, 715 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-13T15:38:25.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 179594 3.99 MB/sec execute 121 sec latency 12.340 ms 2024-09-13T15:38:25.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 547751 12.82 MB/sec execute 121 sec latency 38.258 ms 2024-09-13T15:38:26.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 180648 4.03 MB/sec execute 122 sec latency 12.952 ms 2024-09-13T15:38:26.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 548710 12.75 MB/sec execute 122 sec latency 294.299 ms 2024-09-13T15:38:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:25 smithi090 ceph-mon[27811]: pgmap v393: 129 pgs: 129 active+clean; 89 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 116 op/s 2024-09-13T15:38:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:25 smithi175 ceph-mon[36128]: pgmap v393: 129 pgs: 129 active+clean; 89 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 116 op/s 2024-09-13T15:38:27.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 181253 4.04 MB/sec execute 123 sec latency 10.091 ms 2024-09-13T15:38:27.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 551603 12.77 MB/sec execute 123 sec latency 33.330 ms 2024-09-13T15:38:28.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 181949 4.02 MB/sec execute 124 sec latency 11.128 ms 2024-09-13T15:38:28.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 554902 12.82 MB/sec execute 124 sec latency 134.149 ms 2024-09-13T15:38:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:27 smithi090 ceph-mon[27811]: pgmap v394: 129 pgs: 129 active+clean; 88 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-09-13T15:38:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:27 smithi175 ceph-mon[36128]: pgmap v394: 129 pgs: 129 active+clean; 88 MiB data, 720 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 91 op/s 2024-09-13T15:38:29.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 182777 4.02 MB/sec execute 125 sec latency 21.120 ms 2024-09-13T15:38:29.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 556928 12.80 MB/sec execute 125 sec latency 293.625 ms 2024-09-13T15:38:30.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 183512 4.00 MB/sec execute 126 sec latency 74.164 ms 2024-09-13T15:38:30.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 559193 12.80 MB/sec execute 126 sec latency 175.331 ms 2024-09-13T15:38:30.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:29 smithi090 ceph-mon[27811]: pgmap v395: 129 pgs: 129 active+clean; 89 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.2 MiB/s wr, 101 op/s 2024-09-13T15:38:30.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:29 smithi175 ceph-mon[36128]: pgmap v395: 129 pgs: 129 active+clean; 89 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.2 MiB/s wr, 101 op/s 2024-09-13T15:38:31.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 184421 4.05 MB/sec execute 127 sec latency 10.509 ms 2024-09-13T15:38:31.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 561993 12.81 MB/sec execute 127 sec latency 157.720 ms 2024-09-13T15:38:32.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 185016 4.03 MB/sec execute 128 sec latency 9.865 ms 2024-09-13T15:38:32.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 565527 12.86 MB/sec execute 128 sec latency 33.430 ms 2024-09-13T15:38:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:31 smithi090 ceph-mon[27811]: pgmap v396: 129 pgs: 129 active+clean; 90 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-13T15:38:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:31 smithi175 ceph-mon[36128]: pgmap v396: 129 pgs: 129 active+clean; 90 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-13T15:38:33.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 185841 4.04 MB/sec execute 129 sec latency 11.611 ms 2024-09-13T15:38:33.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 569021 12.91 MB/sec execute 129 sec latency 41.784 ms 2024-09-13T15:38:34.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 186715 4.01 MB/sec execute 130 sec latency 17.471 ms 2024-09-13T15:38:34.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 571644 12.92 MB/sec execute 130 sec latency 34.349 ms 2024-09-13T15:38:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:33 smithi090 ceph-mon[27811]: pgmap v397: 129 pgs: 129 active+clean; 90 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 92 op/s 2024-09-13T15:38:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:33 smithi175 ceph-mon[36128]: pgmap v397: 129 pgs: 129 active+clean; 90 MiB data, 732 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 92 op/s 2024-09-13T15:38:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:35.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 187622 4.04 MB/sec execute 131 sec latency 62.256 ms 2024-09-13T15:38:35.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572620 12.86 MB/sec execute 131 sec latency 334.768 ms 2024-09-13T15:38:36.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 188097 4.04 MB/sec execute 132 sec latency 135.650 ms 2024-09-13T15:38:36.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572627 12.76 MB/sec execute 132 sec latency 370.455 ms 2024-09-13T15:38:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:35 smithi090 ceph-mon[27811]: pgmap v398: 129 pgs: 129 active+clean; 92 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 117 op/s 2024-09-13T15:38:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:35 smithi175 ceph-mon[36128]: pgmap v398: 129 pgs: 129 active+clean; 92 MiB data, 730 MiB used, 715 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 117 op/s 2024-09-13T15:38:37.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 188477 4.02 MB/sec execute 133 sec latency 126.603 ms 2024-09-13T15:38:37.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572688 12.67 MB/sec execute 133 sec latency 450.676 ms 2024-09-13T15:38:38.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 188986 4.00 MB/sec execute 134 sec latency 101.804 ms 2024-09-13T15:38:38.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572695 12.57 MB/sec execute 134 sec latency 376.779 ms 2024-09-13T15:38:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:37 smithi090 ceph-mon[27811]: pgmap v399: 129 pgs: 129 active+clean; 91 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:38:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:37 smithi175 ceph-mon[36128]: pgmap v399: 129 pgs: 129 active+clean; 91 MiB data, 746 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:38:39.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 189619 4.00 MB/sec execute 135 sec latency 136.712 ms 2024-09-13T15:38:39.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572700 12.48 MB/sec execute 135 sec latency 361.838 ms 2024-09-13T15:38:40.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 190370 3.98 MB/sec execute 136 sec latency 77.421 ms 2024-09-13T15:38:40.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572811 12.39 MB/sec execute 136 sec latency 377.937 ms 2024-09-13T15:38:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:39 smithi090 ceph-mon[27811]: pgmap v400: 129 pgs: 129 active+clean; 92 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-09-13T15:38:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:39 smithi175 ceph-mon[36128]: pgmap v400: 129 pgs: 129 active+clean; 92 MiB data, 749 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-09-13T15:38:41.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 191192 4.00 MB/sec execute 137 sec latency 117.692 ms 2024-09-13T15:38:41.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 572880 12.30 MB/sec execute 137 sec latency 367.071 ms 2024-09-13T15:38:42.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 191570 4.00 MB/sec execute 138 sec latency 133.137 ms 2024-09-13T15:38:42.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 573527 12.24 MB/sec execute 138 sec latency 427.422 ms 2024-09-13T15:38:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:41 smithi090 ceph-mon[27811]: pgmap v401: 129 pgs: 129 active+clean; 93 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-13T15:38:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:41 smithi175 ceph-mon[36128]: pgmap v401: 129 pgs: 129 active+clean; 93 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-13T15:38:43.009 INFO:tasks.workunit.client.1.smithi175.stdout: 1 192034 3.98 MB/sec execute 139 sec latency 122.598 ms 2024-09-13T15:38:43.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 573782 12.16 MB/sec execute 139 sec latency 375.740 ms 2024-09-13T15:38:44.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 192601 3.97 MB/sec execute 140 sec latency 170.558 ms 2024-09-13T15:38:44.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 574265 12.08 MB/sec execute 140 sec latency 423.628 ms 2024-09-13T15:38:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:43 smithi090 ceph-mon[27811]: pgmap v402: 129 pgs: 129 active+clean; 94 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:38:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:43 smithi175 ceph-mon[36128]: pgmap v402: 129 pgs: 129 active+clean; 94 MiB data, 770 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:38:45.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 193394 3.97 MB/sec execute 141 sec latency 133.763 ms 2024-09-13T15:38:45.204 INFO:tasks.workunit.client.0.smithi090.stdout: 1 576246 12.08 MB/sec execute 141 sec latency 359.131 ms 2024-09-13T15:38:46.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 194164 3.96 MB/sec execute 142 sec latency 13.511 ms 2024-09-13T15:38:46.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 579722 12.13 MB/sec execute 142 sec latency 75.216 ms 2024-09-13T15:38:46.272 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:45 smithi090 ceph-mon[27811]: pgmap v403: 129 pgs: 129 active+clean; 94 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-13T15:38:46.273 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:45 smithi175 ceph-mon[36128]: pgmap v403: 129 pgs: 129 active+clean; 94 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-13T15:38:47.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 195105 4.00 MB/sec execute 143 sec latency 10.977 ms 2024-09-13T15:38:47.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 583264 12.18 MB/sec execute 143 sec latency 31.124 ms 2024-09-13T15:38:48.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 195667 3.98 MB/sec execute 144 sec latency 11.028 ms 2024-09-13T15:38:48.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 586754 12.23 MB/sec execute 144 sec latency 60.936 ms 2024-09-13T15:38:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:47 smithi090 ceph-mon[27811]: pgmap v404: 129 pgs: 129 active+clean; 94 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 68 op/s 2024-09-13T15:38:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:47 smithi175 ceph-mon[36128]: pgmap v404: 129 pgs: 129 active+clean; 94 MiB data, 788 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.4 MiB/s wr, 68 op/s 2024-09-13T15:38:49.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 196500 3.99 MB/sec execute 145 sec latency 11.568 ms 2024-09-13T15:38:49.167 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:49.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 589409 12.24 MB/sec execute 145 sec latency 33.509 ms 2024-09-13T15:38:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:38:50.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 197390 3.97 MB/sec execute 146 sec latency 11.468 ms 2024-09-13T15:38:50.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 590613 12.20 MB/sec execute 146 sec latency 314.361 ms 2024-09-13T15:38:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:49 smithi090 ceph-mon[27811]: pgmap v405: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-13T15:38:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:49 smithi175 ceph-mon[36128]: pgmap v405: 129 pgs: 129 active+clean; 94 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 94 op/s 2024-09-13T15:38:51.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 197994 3.98 MB/sec execute 147 sec latency 145.334 ms 2024-09-13T15:38:51.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 591260 12.14 MB/sec execute 147 sec latency 369.526 ms 2024-09-13T15:38:52.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 198484 3.97 MB/sec execute 148 sec latency 141.560 ms 2024-09-13T15:38:52.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 593843 12.16 MB/sec execute 148 sec latency 377.918 ms 2024-09-13T15:38:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:51 smithi090 ceph-mon[27811]: pgmap v406: 129 pgs: 129 active+clean; 94 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:38:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:51 smithi175 ceph-mon[36128]: pgmap v406: 129 pgs: 129 active+clean; 94 MiB data, 799 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 MiB/s wr, 80 op/s 2024-09-13T15:38:53.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 199147 3.97 MB/sec execute 149 sec latency 45.138 ms 2024-09-13T15:38:53.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 596467 12.17 MB/sec execute 149 sec latency 39.062 ms 2024-09-13T15:38:54.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 199909 3.96 MB/sec execute 150 sec latency 10.982 ms 2024-09-13T15:38:54.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 599632 12.20 MB/sec execute 150 sec latency 41.806 ms 2024-09-13T15:38:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:53 smithi090 ceph-mon[27811]: pgmap v407: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-13T15:38:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:53 smithi175 ceph-mon[36128]: pgmap v407: 129 pgs: 129 active+clean; 95 MiB data, 801 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-13T15:38:55.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 200823 3.96 MB/sec execute 151 sec latency 17.122 ms 2024-09-13T15:38:55.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 602632 12.21 MB/sec execute 151 sec latency 33.413 ms 2024-09-13T15:38:56.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 201845 3.98 MB/sec execute 152 sec latency 16.853 ms 2024-09-13T15:38:56.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 605717 12.25 MB/sec execute 152 sec latency 91.376 ms 2024-09-13T15:38:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:55 smithi090 ceph-mon[27811]: pgmap v408: 129 pgs: 129 active+clean; 96 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:38:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:55 smithi175 ceph-mon[36128]: pgmap v408: 129 pgs: 129 active+clean; 96 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:38:57.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 202479 3.99 MB/sec execute 153 sec latency 12.064 ms 2024-09-13T15:38:57.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 608152 12.26 MB/sec execute 153 sec latency 194.566 ms 2024-09-13T15:38:58.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 203009 3.97 MB/sec execute 154 sec latency 83.521 ms 2024-09-13T15:38:58.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 608228 12.18 MB/sec execute 154 sec latency 358.719 ms 2024-09-13T15:38:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:57 smithi090 ceph-mon[27811]: pgmap v409: 129 pgs: 129 active+clean; 95 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-13T15:38:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:57 smithi175 ceph-mon[36128]: pgmap v409: 129 pgs: 129 active+clean; 95 MiB data, 811 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-13T15:38:59.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 203611 3.98 MB/sec execute 155 sec latency 132.299 ms 2024-09-13T15:38:59.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 608271 12.11 MB/sec execute 155 sec latency 368.329 ms 2024-09-13T15:39:00.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 204365 3.96 MB/sec execute 156 sec latency 156.028 ms 2024-09-13T15:39:00.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 611627 12.15 MB/sec execute 156 sec latency 373.954 ms 2024-09-13T15:39:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:38:59 smithi090 ceph-mon[27811]: pgmap v410: 129 pgs: 129 active+clean; 97 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-09-13T15:39:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:38:59 smithi175 ceph-mon[36128]: pgmap v410: 129 pgs: 129 active+clean; 97 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 106 op/s 2024-09-13T15:39:01.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 205389 3.98 MB/sec execute 157 sec latency 13.508 ms 2024-09-13T15:39:01.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 614218 12.16 MB/sec execute 157 sec latency 101.809 ms 2024-09-13T15:39:02.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 205995 3.99 MB/sec execute 158 sec latency 11.920 ms 2024-09-13T15:39:02.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 617739 12.20 MB/sec execute 158 sec latency 33.013 ms 2024-09-13T15:39:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:01 smithi090 ceph-mon[27811]: pgmap v411: 129 pgs: 129 active+clean; 97 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:39:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:01 smithi175 ceph-mon[36128]: pgmap v411: 129 pgs: 129 active+clean; 97 MiB data, 825 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:39:03.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 206594 3.97 MB/sec execute 159 sec latency 11.836 ms 2024-09-13T15:39:03.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 620965 12.24 MB/sec execute 159 sec latency 33.447 ms 2024-09-13T15:39:04.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 207424 3.97 MB/sec execute 160 sec latency 11.096 ms 2024-09-13T15:39:04.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 624445 12.28 MB/sec execute 160 sec latency 30.849 ms 2024-09-13T15:39:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:03 smithi090 ceph-mon[27811]: pgmap v412: 129 pgs: 129 active+clean; 96 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:39:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:03 smithi175 ceph-mon[36128]: pgmap v412: 129 pgs: 129 active+clean; 96 MiB data, 824 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:39:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:05.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 208170 3.96 MB/sec execute 161 sec latency 126.293 ms 2024-09-13T15:39:05.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 625818 12.25 MB/sec execute 161 sec latency 339.409 ms 2024-09-13T15:39:06.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 209037 3.98 MB/sec execute 162 sec latency 89.047 ms 2024-09-13T15:39:06.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 625879 12.17 MB/sec execute 162 sec latency 364.292 ms 2024-09-13T15:39:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:05 smithi090 ceph-mon[27811]: pgmap v413: 129 pgs: 129 active+clean; 99 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 106 op/s 2024-09-13T15:39:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:05 smithi175 ceph-mon[36128]: pgmap v413: 129 pgs: 129 active+clean; 99 MiB data, 855 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 106 op/s 2024-09-13T15:39:07.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 209660 3.98 MB/sec execute 163 sec latency 50.195 ms 2024-09-13T15:39:07.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 627842 12.16 MB/sec execute 163 sec latency 343.229 ms 2024-09-13T15:39:08.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 210375 3.97 MB/sec execute 164 sec latency 10.941 ms 2024-09-13T15:39:08.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 630255 12.17 MB/sec execute 164 sec latency 157.036 ms 2024-09-13T15:39:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:07 smithi090 ceph-mon[27811]: pgmap v414: 129 pgs: 129 active+clean; 100 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-13T15:39:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:07 smithi175 ceph-mon[36128]: pgmap v414: 129 pgs: 129 active+clean; 100 MiB data, 856 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-13T15:39:09.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 211247 3.97 MB/sec execute 165 sec latency 17.422 ms 2024-09-13T15:39:09.205 INFO:tasks.workunit.client.0.smithi090.stdout: 1 633156 12.18 MB/sec execute 165 sec latency 27.792 ms 2024-09-13T15:39:10.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 212185 3.99 MB/sec execute 166 sec latency 16.882 ms 2024-09-13T15:39:10.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 636633 12.22 MB/sec execute 166 sec latency 33.216 ms 2024-09-13T15:39:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:09 smithi090 ceph-mon[27811]: pgmap v415: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 104 op/s 2024-09-13T15:39:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:09 smithi175 ceph-mon[36128]: pgmap v415: 129 pgs: 129 active+clean; 100 MiB data, 859 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 104 op/s 2024-09-13T15:39:11.010 INFO:tasks.workunit.client.1.smithi175.stdout: 1 212961 4.00 MB/sec execute 167 sec latency 11.896 ms 2024-09-13T15:39:11.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 639998 12.26 MB/sec execute 167 sec latency 102.474 ms 2024-09-13T15:39:12.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 213561 3.98 MB/sec execute 168 sec latency 12.252 ms 2024-09-13T15:39:12.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 641735 12.23 MB/sec execute 168 sec latency 313.866 ms 2024-09-13T15:39:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:11 smithi090 ceph-mon[27811]: pgmap v416: 129 pgs: 129 active+clean; 101 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:39:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:11 smithi175 ceph-mon[36128]: pgmap v416: 129 pgs: 129 active+clean; 101 MiB data, 860 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:39:13.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 214308 3.99 MB/sec execute 169 sec latency 74.093 ms 2024-09-13T15:39:13.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 642237 12.19 MB/sec execute 169 sec latency 404.851 ms 2024-09-13T15:39:14.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 215077 3.97 MB/sec execute 170 sec latency 85.106 ms 2024-09-13T15:39:14.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 643539 12.16 MB/sec execute 170 sec latency 371.741 ms 2024-09-13T15:39:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:13 smithi090 ceph-mon[27811]: pgmap v417: 129 pgs: 129 active+clean; 103 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-13T15:39:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:13 smithi175 ceph-mon[36128]: pgmap v417: 129 pgs: 129 active+clean; 103 MiB data, 862 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 96 op/s 2024-09-13T15:39:15.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 216157 3.99 MB/sec execute 171 sec latency 12.005 ms 2024-09-13T15:39:15.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 647026 12.20 MB/sec execute 171 sec latency 148.074 ms 2024-09-13T15:39:16.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 216759 4.00 MB/sec execute 172 sec latency 11.847 ms 2024-09-13T15:39:16.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 649329 12.20 MB/sec execute 172 sec latency 108.718 ms 2024-09-13T15:39:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:15 smithi090 ceph-mon[27811]: pgmap v418: 129 pgs: 129 active+clean; 101 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:39:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:15 smithi175 ceph-mon[36128]: pgmap v418: 129 pgs: 129 active+clean; 101 MiB data, 867 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:39:17.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 217391 3.98 MB/sec execute 173 sec latency 13.833 ms 2024-09-13T15:39:17.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 651531 12.20 MB/sec execute 173 sec latency 138.565 ms 2024-09-13T15:39:18.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 218254 3.98 MB/sec execute 174 sec latency 11.902 ms 2024-09-13T15:39:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:17 smithi175 ceph-mon[36128]: pgmap v419: 129 pgs: 129 active+clean; 102 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-13T15:39:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:39:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:39:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:39:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:17 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:39:18.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 654367 12.21 MB/sec execute 174 sec latency 33.556 ms 2024-09-13T15:39:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:17 smithi090 ceph-mon[27811]: pgmap v419: 129 pgs: 129 active+clean; 102 MiB data, 873 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-13T15:39:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:39:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:39:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:39:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:17 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:39:19.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 219041 3.98 MB/sec execute 175 sec latency 13.336 ms 2024-09-13T15:39:19.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 657795 12.25 MB/sec execute 175 sec latency 33.486 ms 2024-09-13T15:39:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:20.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 219949 4.01 MB/sec execute 176 sec latency 11.016 ms 2024-09-13T15:39:20.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 661274 12.29 MB/sec execute 176 sec latency 78.014 ms 2024-09-13T15:39:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:19 smithi090 ceph-mon[27811]: pgmap v420: 129 pgs: 129 active+clean; 103 MiB data, 875 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:39:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:19 smithi175 ceph-mon[36128]: pgmap v420: 129 pgs: 129 active+clean; 103 MiB data, 875 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:39:21.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 220562 3.99 MB/sec execute 177 sec latency 9.352 ms 2024-09-13T15:39:21.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 664762 12.33 MB/sec execute 177 sec latency 35.073 ms 2024-09-13T15:39:22.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 221416 4.00 MB/sec execute 178 sec latency 12.108 ms 2024-09-13T15:39:22.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 668299 12.36 MB/sec execute 178 sec latency 48.551 ms 2024-09-13T15:39:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:21 smithi090 ceph-mon[27811]: pgmap v421: 129 pgs: 129 active+clean; 103 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:39:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:21 smithi175 ceph-mon[36128]: pgmap v421: 129 pgs: 129 active+clean; 103 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:39:23.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 222301 3.99 MB/sec execute 179 sec latency 12.507 ms 2024-09-13T15:39:23.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 670816 12.37 MB/sec execute 179 sec latency 33.403 ms 2024-09-13T15:39:24.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 223260 4.01 MB/sec execute 180 sec latency 10.713 ms 2024-09-13T15:39:24.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 674138 12.40 MB/sec execute 180 sec latency 25.842 ms 2024-09-13T15:39:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:23 smithi090 ceph-mon[27811]: pgmap v422: 129 pgs: 129 active+clean; 104 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-13T15:39:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:23 smithi175 ceph-mon[36128]: pgmap v422: 129 pgs: 129 active+clean; 104 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 103 op/s 2024-09-13T15:39:25.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 223853 4.01 MB/sec execute 181 sec latency 12.093 ms 2024-09-13T15:39:25.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 677113 12.41 MB/sec execute 181 sec latency 33.032 ms 2024-09-13T15:39:26.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 224517 3.99 MB/sec execute 182 sec latency 10.465 ms 2024-09-13T15:39:26.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 679302 12.41 MB/sec execute 182 sec latency 100.409 ms 2024-09-13T15:39:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:25 smithi090 ceph-mon[27811]: pgmap v423: 129 pgs: 129 active+clean; 103 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-13T15:39:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:25 smithi175 ceph-mon[36128]: pgmap v423: 129 pgs: 129 active+clean; 103 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-13T15:39:27.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 225360 4.00 MB/sec execute 183 sec latency 53.634 ms 2024-09-13T15:39:27.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 682667 12.45 MB/sec execute 183 sec latency 175.333 ms 2024-09-13T15:39:28.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 226113 3.99 MB/sec execute 184 sec latency 39.802 ms 2024-09-13T15:39:28.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 686144 12.48 MB/sec execute 184 sec latency 44.558 ms 2024-09-13T15:39:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:27 smithi090 ceph-mon[27811]: pgmap v424: 129 pgs: 129 active+clean; 103 MiB data, 893 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-09-13T15:39:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:27 smithi175 ceph-mon[36128]: pgmap v424: 129 pgs: 129 active+clean; 103 MiB data, 893 MiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.6 MiB/s wr, 87 op/s 2024-09-13T15:39:29.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 227023 4.02 MB/sec execute 185 sec latency 11.149 ms 2024-09-13T15:39:29.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 689683 12.52 MB/sec execute 185 sec latency 33.035 ms 2024-09-13T15:39:30.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 227599 4.01 MB/sec execute 186 sec latency 10.953 ms 2024-09-13T15:39:30.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 693173 12.55 MB/sec execute 186 sec latency 33.398 ms 2024-09-13T15:39:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:29 smithi090 ceph-mon[27811]: pgmap v425: 129 pgs: 129 active+clean; 105 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-09-13T15:39:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:29 smithi175 ceph-mon[36128]: pgmap v425: 129 pgs: 129 active+clean; 105 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 5.6 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-09-13T15:39:31.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 228444 4.01 MB/sec execute 187 sec latency 11.260 ms 2024-09-13T15:39:31.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 696717 12.59 MB/sec execute 187 sec latency 33.368 ms 2024-09-13T15:39:32.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 229279 3.99 MB/sec execute 188 sec latency 11.369 ms 2024-09-13T15:39:32.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 699341 12.60 MB/sec execute 188 sec latency 27.158 ms 2024-09-13T15:39:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:31 smithi090 ceph-mon[27811]: pgmap v426: 129 pgs: 129 active+clean; 105 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:39:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:31 smithi175 ceph-mon[36128]: pgmap v426: 129 pgs: 129 active+clean; 105 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:39:33.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 230335 4.02 MB/sec execute 189 sec latency 10.948 ms 2024-09-13T15:39:33.206 INFO:tasks.workunit.client.0.smithi090.stdout: 1 701961 12.59 MB/sec execute 189 sec latency 112.195 ms 2024-09-13T15:39:34.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 230922 4.02 MB/sec execute 190 sec latency 11.194 ms 2024-09-13T15:39:34.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 705022 12.62 MB/sec execute 190 sec latency 33.507 ms 2024-09-13T15:39:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:33 smithi090 ceph-mon[27811]: pgmap v427: 129 pgs: 129 active+clean; 105 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-13T15:39:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:33 smithi175 ceph-mon[36128]: pgmap v427: 129 pgs: 129 active+clean; 105 MiB data, 885 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-13T15:39:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:35.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 231531 4.01 MB/sec execute 191 sec latency 57.402 ms 2024-09-13T15:39:35.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 707415 12.62 MB/sec execute 191 sec latency 286.411 ms 2024-09-13T15:39:36.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 232372 4.01 MB/sec execute 192 sec latency 11.342 ms 2024-09-13T15:39:36.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 710961 12.66 MB/sec execute 192 sec latency 25.798 ms 2024-09-13T15:39:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:35 smithi090 ceph-mon[27811]: pgmap v428: 129 pgs: 129 active+clean; 106 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 MiB/s wr, 97 op/s 2024-09-13T15:39:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:35 smithi175 ceph-mon[36128]: pgmap v428: 129 pgs: 129 active+clean; 106 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 MiB/s wr, 97 op/s 2024-09-13T15:39:37.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 233187 4.00 MB/sec execute 193 sec latency 18.229 ms 2024-09-13T15:39:37.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 714450 12.69 MB/sec execute 193 sec latency 34.521 ms 2024-09-13T15:39:38.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 233999 4.01 MB/sec execute 194 sec latency 12.572 ms 2024-09-13T15:39:38.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 716143 12.66 MB/sec execute 194 sec latency 158.310 ms 2024-09-13T15:39:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:37 smithi090 ceph-mon[27811]: pgmap v429: 129 pgs: 129 active+clean; 105 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-13T15:39:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:37 smithi175 ceph-mon[36128]: pgmap v429: 129 pgs: 129 active+clean; 105 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 84 op/s 2024-09-13T15:39:39.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 234694 4.02 MB/sec execute 195 sec latency 11.703 ms 2024-09-13T15:39:39.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 718953 12.68 MB/sec execute 195 sec latency 54.223 ms 2024-09-13T15:39:40.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 235530 4.02 MB/sec execute 196 sec latency 34.410 ms 2024-09-13T15:39:40.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 721601 12.69 MB/sec execute 196 sec latency 283.409 ms 2024-09-13T15:39:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:39 smithi090 ceph-mon[27811]: pgmap v430: 129 pgs: 129 active+clean; 107 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:39:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:39 smithi175 ceph-mon[36128]: pgmap v430: 129 pgs: 129 active+clean; 107 MiB data, 907 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.2 MiB/s wr, 106 op/s 2024-09-13T15:39:41.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 236290 4.01 MB/sec execute 197 sec latency 66.355 ms 2024-09-13T15:39:41.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 721662 12.63 MB/sec execute 197 sec latency 369.653 ms 2024-09-13T15:39:42.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 236765 4.00 MB/sec execute 198 sec latency 153.386 ms 2024-09-13T15:39:42.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 721669 12.57 MB/sec execute 198 sec latency 473.590 ms 2024-09-13T15:39:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:41 smithi090 ceph-mon[27811]: pgmap v431: 129 pgs: 129 active+clean; 107 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-13T15:39:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:41 smithi175 ceph-mon[36128]: pgmap v431: 129 pgs: 129 active+clean; 107 MiB data, 908 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-13T15:39:43.011 INFO:tasks.workunit.client.1.smithi175.stdout: 1 237536 4.01 MB/sec execute 199 sec latency 100.393 ms 2024-09-13T15:39:43.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 721743 12.50 MB/sec execute 199 sec latency 380.302 ms 2024-09-13T15:39:44.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 238128 4.01 MB/sec execute 200 sec latency 78.432 ms 2024-09-13T15:39:44.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 721846 12.44 MB/sec execute 200 sec latency 339.814 ms 2024-09-13T15:39:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:43 smithi090 ceph-mon[27811]: pgmap v432: 129 pgs: 129 active+clean; 107 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-13T15:39:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:43 smithi175 ceph-mon[36128]: pgmap v432: 129 pgs: 129 active+clean; 107 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-13T15:39:45.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 238637 4.00 MB/sec execute 201 sec latency 146.033 ms 2024-09-13T15:39:45.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 722494 12.40 MB/sec execute 201 sec latency 372.688 ms 2024-09-13T15:39:46.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 239315 4.00 MB/sec execute 202 sec latency 127.529 ms 2024-09-13T15:39:46.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 722749 12.34 MB/sec execute 202 sec latency 376.965 ms 2024-09-13T15:39:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:45 smithi090 ceph-mon[27811]: pgmap v433: 129 pgs: 129 active+clean; 110 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 112 op/s 2024-09-13T15:39:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:45 smithi175 ceph-mon[36128]: pgmap v433: 129 pgs: 129 active+clean; 110 MiB data, 920 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 112 op/s 2024-09-13T15:39:47.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 239992 3.99 MB/sec execute 203 sec latency 141.449 ms 2024-09-13T15:39:47.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 723234 12.29 MB/sec execute 203 sec latency 371.232 ms 2024-09-13T15:39:48.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 240700 4.00 MB/sec execute 204 sec latency 135.719 ms 2024-09-13T15:39:48.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 723622 12.25 MB/sec execute 204 sec latency 358.435 ms 2024-09-13T15:39:48.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:47 smithi090 ceph-mon[27811]: pgmap v434: 129 pgs: 129 active+clean; 110 MiB data, 922 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:39:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:47 smithi175 ceph-mon[36128]: pgmap v434: 129 pgs: 129 active+clean; 110 MiB data, 922 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:39:49.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 241384 4.00 MB/sec execute 205 sec latency 141.578 ms 2024-09-13T15:39:49.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 723853 12.20 MB/sec execute 205 sec latency 468.799 ms 2024-09-13T15:39:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:48 smithi090 ceph-mon[27811]: pgmap v435: 129 pgs: 129 active+clean; 110 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-09-13T15:39:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:39:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:48 smithi175 ceph-mon[36128]: pgmap v435: 129 pgs: 129 active+clean; 110 MiB data, 926 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.1 MiB/s wr, 88 op/s 2024-09-13T15:39:50.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 241774 3.99 MB/sec execute 206 sec latency 116.147 ms 2024-09-13T15:39:50.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 724150 12.14 MB/sec execute 206 sec latency 438.891 ms 2024-09-13T15:39:51.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 242415 3.98 MB/sec execute 207 sec latency 137.837 ms 2024-09-13T15:39:51.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 725892 12.13 MB/sec execute 207 sec latency 340.664 ms 2024-09-13T15:39:52.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:51 smithi090 ceph-mon[27811]: pgmap v436: 129 pgs: 129 active+clean; 111 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:39:52.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 243322 3.98 MB/sec execute 208 sec latency 14.088 ms 2024-09-13T15:39:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:51 smithi175 ceph-mon[36128]: pgmap v436: 129 pgs: 129 active+clean; 111 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:39:52.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 728833 12.15 MB/sec execute 208 sec latency 153.395 ms 2024-09-13T15:39:53.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 244257 3.99 MB/sec execute 209 sec latency 10.959 ms 2024-09-13T15:39:53.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 732301 12.18 MB/sec execute 209 sec latency 51.782 ms 2024-09-13T15:39:54.006 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:53 smithi090 ceph-mon[27811]: pgmap v437: 129 pgs: 129 active+clean; 111 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 74 op/s 2024-09-13T15:39:54.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 245043 4.00 MB/sec execute 210 sec latency 11.845 ms 2024-09-13T15:39:54.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:53 smithi175 ceph-mon[36128]: pgmap v437: 129 pgs: 129 active+clean; 111 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 74 op/s 2024-09-13T15:39:54.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 735779 12.21 MB/sec execute 210 sec latency 48.213 ms 2024-09-13T15:39:55.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 245659 3.99 MB/sec execute 211 sec latency 10.598 ms 2024-09-13T15:39:55.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 739324 12.24 MB/sec execute 211 sec latency 33.409 ms 2024-09-13T15:39:56.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:55 smithi090 ceph-mon[27811]: pgmap v438: 129 pgs: 129 active+clean; 111 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-13T15:39:56.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 246409 3.99 MB/sec execute 212 sec latency 124.288 ms 2024-09-13T15:39:56.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:55 smithi175 ceph-mon[36128]: pgmap v438: 129 pgs: 129 active+clean; 111 MiB data, 904 MiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.8 MiB/s wr, 91 op/s 2024-09-13T15:39:56.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 741003 12.22 MB/sec execute 212 sec latency 369.321 ms 2024-09-13T15:39:57.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 247135 3.98 MB/sec execute 213 sec latency 121.863 ms 2024-09-13T15:39:57.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 741505 12.19 MB/sec execute 213 sec latency 365.652 ms 2024-09-13T15:39:58.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:57 smithi090 ceph-mon[27811]: pgmap v439: 129 pgs: 129 active+clean; 111 MiB data, 906 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-09-13T15:39:58.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 248050 4.00 MB/sec execute 214 sec latency 95.605 ms 2024-09-13T15:39:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:57 smithi175 ceph-mon[36128]: pgmap v439: 129 pgs: 129 active+clean; 111 MiB data, 906 MiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.3 MiB/s wr, 67 op/s 2024-09-13T15:39:58.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 741911 12.14 MB/sec execute 214 sec latency 368.123 ms 2024-09-13T15:39:59.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 248609 4.00 MB/sec execute 215 sec latency 93.461 ms 2024-09-13T15:39:59.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 744895 12.15 MB/sec execute 215 sec latency 75.226 ms 2024-09-13T15:40:00.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:39:59 smithi090 ceph-mon[27811]: pgmap v440: 129 pgs: 129 active+clean; 113 MiB data, 906 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 93 op/s 2024-09-13T15:40:00.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 249243 3.98 MB/sec execute 216 sec latency 11.810 ms 2024-09-13T15:40:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:39:59 smithi175 ceph-mon[36128]: pgmap v440: 129 pgs: 129 active+clean; 113 MiB data, 906 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 93 op/s 2024-09-13T15:40:00.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 748023 12.17 MB/sec execute 216 sec latency 33.263 ms 2024-09-13T15:40:01.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:00 smithi090 ceph-mon[27811]: overall HEALTH_OK 2024-09-13T15:40:01.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 250118 3.99 MB/sec execute 217 sec latency 14.506 ms 2024-09-13T15:40:01.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:00 smithi175 ceph-mon[36128]: overall HEALTH_OK 2024-09-13T15:40:01.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 750707 12.19 MB/sec execute 217 sec latency 64.741 ms 2024-09-13T15:40:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:01 smithi090 ceph-mon[27811]: pgmap v441: 129 pgs: 129 active+clean; 113 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-13T15:40:02.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 250930 3.98 MB/sec execute 218 sec latency 9.921 ms 2024-09-13T15:40:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:01 smithi175 ceph-mon[36128]: pgmap v441: 129 pgs: 129 active+clean; 113 MiB data, 911 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 85 op/s 2024-09-13T15:40:02.207 INFO:tasks.workunit.client.0.smithi090.stdout: 1 753692 12.20 MB/sec execute 218 sec latency 33.365 ms 2024-09-13T15:40:03.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 251838 4.01 MB/sec execute 219 sec latency 38.819 ms 2024-09-13T15:40:03.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 757052 12.23 MB/sec execute 219 sec latency 151.319 ms 2024-09-13T15:40:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:03 smithi090 ceph-mon[27811]: pgmap v442: 129 pgs: 129 active+clean; 114 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:40:04.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:04.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 252271 4.00 MB/sec execute 220 sec latency 151.101 ms 2024-09-13T15:40:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:03 smithi175 ceph-mon[36128]: pgmap v442: 129 pgs: 129 active+clean; 114 MiB data, 934 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:40:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:04.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 757057 12.18 MB/sec execute 220 sec latency 408.545 ms 2024-09-13T15:40:05.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 252859 3.98 MB/sec execute 221 sec latency 161.296 ms 2024-09-13T15:40:05.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 757118 12.12 MB/sec execute 221 sec latency 481.990 ms 2024-09-13T15:40:06.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:05 smithi090 ceph-mon[27811]: pgmap v443: 129 pgs: 129 active+clean; 114 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-09-13T15:40:06.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 253708 3.98 MB/sec execute 222 sec latency 72.320 ms 2024-09-13T15:40:06.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:05 smithi175 ceph-mon[36128]: pgmap v443: 129 pgs: 129 active+clean; 114 MiB data, 938 MiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-09-13T15:40:06.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 760604 12.15 MB/sec execute 222 sec latency 294.181 ms 2024-09-13T15:40:07.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 254494 3.98 MB/sec execute 223 sec latency 14.009 ms 2024-09-13T15:40:07.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 763061 12.16 MB/sec execute 223 sec latency 133.433 ms 2024-09-13T15:40:08.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:07 smithi090 ceph-mon[27811]: pgmap v444: 129 pgs: 129 active+clean; 115 MiB data, 943 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-13T15:40:08.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 255428 4.00 MB/sec execute 224 sec latency 10.751 ms 2024-09-13T15:40:08.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:07 smithi175 ceph-mon[36128]: pgmap v444: 129 pgs: 129 active+clean; 115 MiB data, 943 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 77 op/s 2024-09-13T15:40:08.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 765049 12.15 MB/sec execute 224 sec latency 141.704 ms 2024-09-13T15:40:09.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 256030 3.99 MB/sec execute 225 sec latency 8.584 ms 2024-09-13T15:40:09.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 767701 12.16 MB/sec execute 225 sec latency 158.321 ms 2024-09-13T15:40:10.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:09 smithi090 ceph-mon[27811]: pgmap v445: 129 pgs: 129 active+clean; 116 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 104 op/s 2024-09-13T15:40:10.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 256892 4.00 MB/sec execute 226 sec latency 8.872 ms 2024-09-13T15:40:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:09 smithi175 ceph-mon[36128]: pgmap v445: 129 pgs: 129 active+clean; 116 MiB data, 944 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 104 op/s 2024-09-13T15:40:10.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 769724 12.15 MB/sec execute 226 sec latency 289.773 ms 2024-09-13T15:40:11.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 257760 3.99 MB/sec execute 227 sec latency 16.991 ms 2024-09-13T15:40:11.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 772137 12.16 MB/sec execute 227 sec latency 166.759 ms 2024-09-13T15:40:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:11 smithi090 ceph-mon[27811]: pgmap v446: 129 pgs: 129 active+clean; 115 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-13T15:40:12.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 258718 4.00 MB/sec execute 228 sec latency 12.052 ms 2024-09-13T15:40:12.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:11 smithi175 ceph-mon[36128]: pgmap v446: 129 pgs: 129 active+clean; 115 MiB data, 946 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-13T15:40:12.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 774793 12.17 MB/sec execute 228 sec latency 166.677 ms 2024-09-13T15:40:13.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 259342 4.01 MB/sec execute 229 sec latency 10.713 ms 2024-09-13T15:40:13.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 777383 12.18 MB/sec execute 229 sec latency 158.475 ms 2024-09-13T15:40:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:13 smithi090 ceph-mon[27811]: pgmap v447: 129 pgs: 129 active+clean; 116 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-13T15:40:14.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 260025 4.00 MB/sec execute 230 sec latency 14.074 ms 2024-09-13T15:40:14.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:13 smithi175 ceph-mon[36128]: pgmap v447: 129 pgs: 129 active+clean; 116 MiB data, 928 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-13T15:40:14.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 779968 12.18 MB/sec execute 230 sec latency 133.305 ms 2024-09-13T15:40:15.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 260887 3.99 MB/sec execute 231 sec latency 16.107 ms 2024-09-13T15:40:15.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 782627 12.20 MB/sec execute 231 sec latency 75.076 ms 2024-09-13T15:40:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:15 smithi090 ceph-mon[27811]: pgmap v448: 129 pgs: 129 active+clean; 118 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-09-13T15:40:16.012 INFO:tasks.workunit.client.1.smithi175.stdout: 1 261846 4.00 MB/sec execute 232 sec latency 8.977 ms 2024-09-13T15:40:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:15 smithi175 ceph-mon[36128]: pgmap v448: 129 pgs: 129 active+clean; 118 MiB data, 950 MiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 94 op/s 2024-09-13T15:40:16.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 785608 12.21 MB/sec execute 232 sec latency 33.515 ms 2024-09-13T15:40:17.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 262651 4.01 MB/sec execute 233 sec latency 12.429 ms 2024-09-13T15:40:17.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 789032 12.24 MB/sec execute 233 sec latency 140.568 ms 2024-09-13T15:40:18.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:17 smithi090 ceph-mon[27811]: pgmap v449: 129 pgs: 129 active+clean; 118 MiB data, 955 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-13T15:40:18.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 263221 4.00 MB/sec execute 234 sec latency 11.127 ms 2024-09-13T15:40:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:17 smithi175 ceph-mon[36128]: pgmap v449: 129 pgs: 129 active+clean; 118 MiB data, 955 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-13T15:40:18.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 792510 12.27 MB/sec execute 234 sec latency 50.572 ms 2024-09-13T15:40:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:40:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:40:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:40:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:40:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:40:19.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:19.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 264018 4.01 MB/sec execute 235 sec latency 13.350 ms 2024-09-13T15:40:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:40:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:40:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:40:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:40:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:40:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:40:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:19.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 795098 12.27 MB/sec execute 235 sec latency 131.544 ms 2024-09-13T15:40:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:19 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi090 to 912.0M 2024-09-13T15:40:20.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:19 smithi090 ceph-mon[27811]: pgmap v450: 129 pgs: 129 active+clean; 120 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-09-13T15:40:20.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 264885 4.00 MB/sec execute 236 sec latency 17.195 ms 2024-09-13T15:40:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:19 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi090 to 912.0M 2024-09-13T15:40:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:19 smithi175 ceph-mon[36128]: pgmap v450: 129 pgs: 129 active+clean; 120 MiB data, 978 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.3 MiB/s wr, 116 op/s 2024-09-13T15:40:20.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 798318 12.29 MB/sec execute 236 sec latency 32.835 ms 2024-09-13T15:40:21.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 265812 4.01 MB/sec execute 237 sec latency 12.653 ms 2024-09-13T15:40:21.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 801630 12.32 MB/sec execute 237 sec latency 31.313 ms 2024-09-13T15:40:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:21 smithi090 ceph-mon[27811]: pgmap v451: 129 pgs: 129 active+clean; 119 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-13T15:40:22.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 266439 4.01 MB/sec execute 238 sec latency 10.660 ms 2024-09-13T15:40:22.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:21 smithi175 ceph-mon[36128]: pgmap v451: 129 pgs: 129 active+clean; 119 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-13T15:40:22.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 804889 12.33 MB/sec execute 238 sec latency 33.092 ms 2024-09-13T15:40:23.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 267153 4.01 MB/sec execute 239 sec latency 8.872 ms 2024-09-13T15:40:23.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 807849 12.35 MB/sec execute 239 sec latency 43.654 ms 2024-09-13T15:40:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:23 smithi090 ceph-mon[27811]: pgmap v452: 129 pgs: 129 active+clean; 119 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-13T15:40:24.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 268046 4.00 MB/sec execute 240 sec latency 13.087 ms 2024-09-13T15:40:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:23 smithi175 ceph-mon[36128]: pgmap v452: 129 pgs: 129 active+clean; 119 MiB data, 979 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 91 op/s 2024-09-13T15:40:24.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 811142 12.38 MB/sec execute 240 sec latency 49.667 ms 2024-09-13T15:40:25.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 268961 4.01 MB/sec execute 241 sec latency 13.260 ms 2024-09-13T15:40:25.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 814034 12.39 MB/sec execute 241 sec latency 33.537 ms 2024-09-13T15:40:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:25 smithi090 ceph-mon[27811]: pgmap v453: 129 pgs: 129 active+clean; 121 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 112 op/s 2024-09-13T15:40:26.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 269727 4.02 MB/sec execute 242 sec latency 45.683 ms 2024-09-13T15:40:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:25 smithi175 ceph-mon[36128]: pgmap v453: 129 pgs: 129 active+clean; 121 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.2 MiB/s wr, 112 op/s 2024-09-13T15:40:26.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 817275 12.41 MB/sec execute 242 sec latency 286.712 ms 2024-09-13T15:40:27.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 270297 4.01 MB/sec execute 243 sec latency 14.676 ms 2024-09-13T15:40:27.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 820111 12.43 MB/sec execute 243 sec latency 33.227 ms 2024-09-13T15:40:28.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:27 smithi090 ceph-mon[27811]: pgmap v454: 129 pgs: 129 active+clean; 122 MiB data, 1007 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:40:28.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 271168 4.02 MB/sec execute 244 sec latency 9.471 ms 2024-09-13T15:40:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:27 smithi175 ceph-mon[36128]: pgmap v454: 129 pgs: 129 active+clean; 122 MiB data, 1007 MiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:40:28.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 822614 12.42 MB/sec execute 244 sec latency 154.383 ms 2024-09-13T15:40:29.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 272000 4.01 MB/sec execute 245 sec latency 13.982 ms 2024-09-13T15:40:29.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 825324 12.44 MB/sec execute 245 sec latency 61.557 ms 2024-09-13T15:40:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:29 smithi090 ceph-mon[27811]: pgmap v455: 129 pgs: 129 active+clean; 123 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.3 MiB/s wr, 112 op/s 2024-09-13T15:40:30.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 272918 4.02 MB/sec execute 246 sec latency 10.924 ms 2024-09-13T15:40:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:29 smithi175 ceph-mon[36128]: pgmap v455: 129 pgs: 129 active+clean; 123 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.3 MiB/s wr, 112 op/s 2024-09-13T15:40:30.208 INFO:tasks.workunit.client.0.smithi090.stdout: 1 828217 12.45 MB/sec execute 246 sec latency 42.032 ms 2024-09-13T15:40:31.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 273567 4.02 MB/sec execute 247 sec latency 12.230 ms 2024-09-13T15:40:31.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 831583 12.47 MB/sec execute 247 sec latency 85.871 ms 2024-09-13T15:40:32.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:31 smithi090 ceph-mon[27811]: pgmap v456: 129 pgs: 129 active+clean; 123 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:40:32.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 274244 4.01 MB/sec execute 248 sec latency 56.175 ms 2024-09-13T15:40:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:31 smithi175 ceph-mon[36128]: pgmap v456: 129 pgs: 129 active+clean; 123 MiB data, 998 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:40:32.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 831659 12.42 MB/sec execute 248 sec latency 332.048 ms 2024-09-13T15:40:33.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 274930 4.01 MB/sec execute 249 sec latency 96.375 ms 2024-09-13T15:40:33.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 831702 12.37 MB/sec execute 249 sec latency 351.610 ms 2024-09-13T15:40:34.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 275709 4.01 MB/sec execute 250 sec latency 41.360 ms 2024-09-13T15:40:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:33 smithi175 ceph-mon[36128]: pgmap v457: 129 pgs: 129 active+clean; 122 MiB data, 997 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 89 op/s 2024-09-13T15:40:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:34.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 832263 12.34 MB/sec execute 250 sec latency 399.901 ms 2024-09-13T15:40:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:33 smithi090 ceph-mon[27811]: pgmap v457: 129 pgs: 129 active+clean; 122 MiB data, 997 MiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 89 op/s 2024-09-13T15:40:34.362 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:35.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 276466 4.02 MB/sec execute 251 sec latency 147.589 ms 2024-09-13T15:40:35.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 832665 12.29 MB/sec execute 251 sec latency 372.077 ms 2024-09-13T15:40:36.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 277011 4.02 MB/sec execute 252 sec latency 119.146 ms 2024-09-13T15:40:36.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:35 smithi175 ceph-mon[36128]: pgmap v458: 129 pgs: 129 active+clean; 124 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.1 MiB/s wr, 112 op/s 2024-09-13T15:40:36.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 832970 12.25 MB/sec execute 252 sec latency 415.843 ms 2024-09-13T15:40:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:35 smithi090 ceph-mon[27811]: pgmap v458: 129 pgs: 129 active+clean; 124 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.1 MiB/s wr, 112 op/s 2024-09-13T15:40:37.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 277584 4.01 MB/sec execute 253 sec latency 65.850 ms 2024-09-13T15:40:37.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 833193 12.20 MB/sec execute 253 sec latency 365.192 ms 2024-09-13T15:40:38.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 278340 4.01 MB/sec execute 254 sec latency 108.652 ms 2024-09-13T15:40:38.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:37 smithi175 ceph-mon[36128]: pgmap v459: 129 pgs: 129 active+clean; 123 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-13T15:40:38.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 833695 12.18 MB/sec execute 254 sec latency 363.681 ms 2024-09-13T15:40:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:37 smithi090 ceph-mon[27811]: pgmap v459: 129 pgs: 129 active+clean; 123 MiB data, 1006 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.6 MiB/s wr, 82 op/s 2024-09-13T15:40:39.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 278988 4.00 MB/sec execute 255 sec latency 141.951 ms 2024-09-13T15:40:39.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 833970 12.13 MB/sec execute 255 sec latency 360.556 ms 2024-09-13T15:40:40.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 279687 4.01 MB/sec execute 256 sec latency 109.822 ms 2024-09-13T15:40:40.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:39 smithi175 ceph-mon[36128]: pgmap v460: 129 pgs: 129 active+clean; 124 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-09-13T15:40:40.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 834102 12.09 MB/sec execute 256 sec latency 375.098 ms 2024-09-13T15:40:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:39 smithi090 ceph-mon[27811]: pgmap v460: 129 pgs: 129 active+clean; 124 MiB data, 1009 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-09-13T15:40:41.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 280322 4.01 MB/sec execute 257 sec latency 113.121 ms 2024-09-13T15:40:41.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 834997 12.06 MB/sec execute 257 sec latency 420.397 ms 2024-09-13T15:40:42.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 280743 4.00 MB/sec execute 258 sec latency 103.608 ms 2024-09-13T15:40:42.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:41 smithi175 ceph-mon[36128]: pgmap v461: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-13T15:40:42.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 835007 12.01 MB/sec execute 258 sec latency 354.511 ms 2024-09-13T15:40:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:41 smithi090 ceph-mon[27811]: pgmap v461: 129 pgs: 129 active+clean; 124 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.5 MiB/s wr, 78 op/s 2024-09-13T15:40:43.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 281526 3.99 MB/sec execute 259 sec latency 12.769 ms 2024-09-13T15:40:43.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 838588 12.04 MB/sec execute 259 sec latency 25.256 ms 2024-09-13T15:40:44.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 282426 3.99 MB/sec execute 260 sec latency 17.264 ms 2024-09-13T15:40:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:43 smithi175 ceph-mon[36128]: pgmap v462: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:40:44.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 841232 12.05 MB/sec execute 260 sec latency 33.510 ms 2024-09-13T15:40:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:43 smithi090 ceph-mon[27811]: pgmap v462: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:40:45.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 283504 4.01 MB/sec execute 261 sec latency 11.212 ms 2024-09-13T15:40:45.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 843850 12.05 MB/sec execute 261 sec latency 152.016 ms 2024-09-13T15:40:46.013 INFO:tasks.workunit.client.1.smithi175.stdout: 1 284087 4.01 MB/sec execute 262 sec latency 11.155 ms 2024-09-13T15:40:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:45 smithi175 ceph-mon[36128]: pgmap v463: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:40:46.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 846660 12.07 MB/sec execute 262 sec latency 61.508 ms 2024-09-13T15:40:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:45 smithi090 ceph-mon[27811]: pgmap v463: 129 pgs: 129 active+clean; 126 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:40:47.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 284723 4.00 MB/sec execute 263 sec latency 10.447 ms 2024-09-13T15:40:47.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 849490 12.08 MB/sec execute 263 sec latency 42.010 ms 2024-09-13T15:40:48.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 285595 4.00 MB/sec execute 264 sec latency 11.046 ms 2024-09-13T15:40:48.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 852918 12.10 MB/sec execute 264 sec latency 141.039 ms 2024-09-13T15:40:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:47 smithi090 ceph-mon[27811]: pgmap v464: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:40:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:47 smithi175 ceph-mon[36128]: pgmap v464: 129 pgs: 129 active+clean; 125 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:40:49.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 286368 4.00 MB/sec execute 265 sec latency 10.884 ms 2024-09-13T15:40:49.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:49.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 856398 12.13 MB/sec execute 265 sec latency 33.336 ms 2024-09-13T15:40:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:40:50.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 287310 4.02 MB/sec execute 266 sec latency 12.574 ms 2024-09-13T15:40:50.044 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:49 smithi090 ceph-mon[27811]: pgmap v465: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:40:50.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 859891 12.16 MB/sec execute 266 sec latency 78.267 ms 2024-09-13T15:40:50.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:49 smithi175 ceph-mon[36128]: pgmap v465: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:40:51.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 287917 4.01 MB/sec execute 267 sec latency 11.050 ms 2024-09-13T15:40:51.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 863421 12.18 MB/sec execute 267 sec latency 41.743 ms 2024-09-13T15:40:52.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 288765 4.01 MB/sec execute 268 sec latency 11.058 ms 2024-09-13T15:40:52.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 866071 12.19 MB/sec execute 268 sec latency 66.317 ms 2024-09-13T15:40:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:51 smithi090 ceph-mon[27811]: pgmap v466: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:40:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:51 smithi175 ceph-mon[36128]: pgmap v466: 129 pgs: 129 active+clean; 127 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:40:53.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 289667 4.00 MB/sec execute 269 sec latency 24.771 ms 2024-09-13T15:40:53.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 869487 12.21 MB/sec execute 269 sec latency 31.404 ms 2024-09-13T15:40:54.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 290629 4.02 MB/sec execute 270 sec latency 10.861 ms 2024-09-13T15:40:54.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 872255 12.21 MB/sec execute 270 sec latency 44.115 ms 2024-09-13T15:40:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:53 smithi090 ceph-mon[27811]: pgmap v467: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:40:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:53 smithi175 ceph-mon[36128]: pgmap v467: 129 pgs: 129 active+clean; 128 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:40:55.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 291262 4.02 MB/sec execute 271 sec latency 10.795 ms 2024-09-13T15:40:55.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 874872 12.23 MB/sec execute 271 sec latency 150.050 ms 2024-09-13T15:40:56.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 291968 4.01 MB/sec execute 272 sec latency 10.261 ms 2024-09-13T15:40:56.209 INFO:tasks.workunit.client.0.smithi090.stdout: 1 877917 12.24 MB/sec execute 272 sec latency 34.561 ms 2024-09-13T15:40:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:55 smithi090 ceph-mon[27811]: pgmap v468: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:40:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:55 smithi175 ceph-mon[36128]: pgmap v468: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:40:57.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 292856 4.01 MB/sec execute 273 sec latency 10.360 ms 2024-09-13T15:40:57.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 881288 12.27 MB/sec execute 273 sec latency 33.531 ms 2024-09-13T15:40:58.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 293790 4.02 MB/sec execute 274 sec latency 17.715 ms 2024-09-13T15:40:58.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 884820 12.29 MB/sec execute 274 sec latency 68.921 ms 2024-09-13T15:40:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:57 smithi090 ceph-mon[27811]: pgmap v469: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:40:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:57 smithi175 ceph-mon[36128]: pgmap v469: 129 pgs: 129 active+clean; 129 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:40:59.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 294621 4.03 MB/sec execute 275 sec latency 12.882 ms 2024-09-13T15:40:59.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 888306 12.31 MB/sec execute 275 sec latency 33.498 ms 2024-09-13T15:41:00.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 295187 4.02 MB/sec execute 276 sec latency 12.691 ms 2024-09-13T15:41:00.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:40:59 smithi175 ceph-mon[36128]: pgmap v470: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-09-13T15:41:00.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 891798 12.34 MB/sec execute 276 sec latency 41.537 ms 2024-09-13T15:41:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:40:59 smithi090 ceph-mon[27811]: pgmap v470: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.2 MiB/s wr, 114 op/s 2024-09-13T15:41:01.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 296034 4.02 MB/sec execute 277 sec latency 14.643 ms 2024-09-13T15:41:01.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 895335 12.36 MB/sec execute 277 sec latency 30.012 ms 2024-09-13T15:41:02.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 296881 4.01 MB/sec execute 278 sec latency 18.141 ms 2024-09-13T15:41:02.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 897959 12.37 MB/sec execute 278 sec latency 33.702 ms 2024-09-13T15:41:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:01 smithi090 ceph-mon[27811]: pgmap v471: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:41:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:01 smithi175 ceph-mon[36128]: pgmap v471: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:41:03.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 297765 4.02 MB/sec execute 279 sec latency 13.321 ms 2024-09-13T15:41:03.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 900091 12.36 MB/sec execute 279 sec latency 211.345 ms 2024-09-13T15:41:04.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 298412 4.03 MB/sec execute 280 sec latency 99.846 ms 2024-09-13T15:41:04.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 902635 12.37 MB/sec execute 280 sec latency 349.881 ms 2024-09-13T15:41:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:03 smithi090 ceph-mon[27811]: pgmap v472: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-13T15:41:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:03 smithi175 ceph-mon[36128]: pgmap v472: 129 pgs: 129 active+clean; 130 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-13T15:41:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:05.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 299143 4.02 MB/sec execute 281 sec latency 11.408 ms 2024-09-13T15:41:05.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 905979 12.39 MB/sec execute 281 sec latency 177.439 ms 2024-09-13T15:41:06.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 300038 4.02 MB/sec execute 282 sec latency 16.582 ms 2024-09-13T15:41:06.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 908585 12.40 MB/sec execute 282 sec latency 125.186 ms 2024-09-13T15:41:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:05 smithi090 ceph-mon[27811]: pgmap v473: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 113 op/s 2024-09-13T15:41:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:05 smithi175 ceph-mon[36128]: pgmap v473: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.1 MiB/s wr, 113 op/s 2024-09-13T15:41:07.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 300961 4.03 MB/sec execute 283 sec latency 18.117 ms 2024-09-13T15:41:07.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 911758 12.41 MB/sec execute 283 sec latency 33.653 ms 2024-09-13T15:41:08.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 301755 4.04 MB/sec execute 284 sec latency 12.069 ms 2024-09-13T15:41:08.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 914729 12.42 MB/sec execute 284 sec latency 33.289 ms 2024-09-13T15:41:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:07 smithi090 ceph-mon[27811]: pgmap v474: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-13T15:41:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:07 smithi175 ceph-mon[36128]: pgmap v474: 129 pgs: 129 active+clean; 132 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-13T15:41:09.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 302303 4.02 MB/sec execute 285 sec latency 55.763 ms 2024-09-13T15:41:09.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 916608 12.42 MB/sec execute 285 sec latency 373.634 ms 2024-09-13T15:41:10.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 302788 4.02 MB/sec execute 286 sec latency 136.476 ms 2024-09-13T15:41:10.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:09 smithi175 ceph-mon[36128]: pgmap v475: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-13T15:41:10.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 916613 12.38 MB/sec execute 286 sec latency 451.172 ms 2024-09-13T15:41:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:09 smithi090 ceph-mon[27811]: pgmap v475: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.3 MiB/s wr, 111 op/s 2024-09-13T15:41:11.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 303594 4.02 MB/sec execute 287 sec latency 61.751 ms 2024-09-13T15:41:11.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917352 12.41 MB/sec execute 287 sec latency 349.194 ms 2024-09-13T15:41:11.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:11 smithi090 ceph-mon[27811]: pgmap v476: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-13T15:41:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:11 smithi175 ceph-mon[36128]: pgmap v476: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-13T15:41:12.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 304290 4.02 MB/sec execute 288 sec latency 154.572 ms 2024-09-13T15:41:12.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917359 12.36 MB/sec execute 288 sec latency 414.056 ms 2024-09-13T15:41:13.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 305072 4.03 MB/sec execute 289 sec latency 138.747 ms 2024-09-13T15:41:13.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917409 12.32 MB/sec execute 289 sec latency 380.590 ms 2024-09-13T15:41:14.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:13 smithi090 ceph-mon[27811]: pgmap v477: 129 pgs: 129 active+clean; 133 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-13T15:41:14.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 305514 4.02 MB/sec execute 290 sec latency 151.680 ms 2024-09-13T15:41:14.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:13 smithi175 ceph-mon[36128]: pgmap v477: 129 pgs: 129 active+clean; 133 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-13T15:41:14.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917414 12.28 MB/sec execute 290 sec latency 371.479 ms 2024-09-13T15:41:15.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 305999 4.01 MB/sec execute 291 sec latency 137.782 ms 2024-09-13T15:41:15.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917421 12.24 MB/sec execute 291 sec latency 364.556 ms 2024-09-13T15:41:16.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:15 smithi090 ceph-mon[27811]: pgmap v478: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:41:16.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 306677 4.01 MB/sec execute 292 sec latency 129.432 ms 2024-09-13T15:41:16.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:15 smithi175 ceph-mon[36128]: pgmap v478: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 107 op/s 2024-09-13T15:41:16.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917482 12.19 MB/sec execute 292 sec latency 360.313 ms 2024-09-13T15:41:17.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 307420 4.00 MB/sec execute 293 sec latency 86.123 ms 2024-09-13T15:41:17.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917489 12.15 MB/sec execute 293 sec latency 358.455 ms 2024-09-13T15:41:18.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:17 smithi090 ceph-mon[27811]: pgmap v479: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-13T15:41:18.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 308074 4.01 MB/sec execute 294 sec latency 157.600 ms 2024-09-13T15:41:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:17 smithi175 ceph-mon[36128]: pgmap v479: 129 pgs: 129 active+clean; 134 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-13T15:41:18.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917601 12.11 MB/sec execute 294 sec latency 381.131 ms 2024-09-13T15:41:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:41:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:19.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 308726 4.01 MB/sec execute 295 sec latency 109.134 ms 2024-09-13T15:41:19.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:41:19.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:19.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 917671 12.07 MB/sec execute 295 sec latency 416.179 ms 2024-09-13T15:41:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:19 smithi090 ceph-mon[27811]: pgmap v480: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:41:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:41:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:41:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:41:20.014 INFO:tasks.workunit.client.1.smithi175.stdout: 1 309182 4.00 MB/sec execute 296 sec latency 114.336 ms 2024-09-13T15:41:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:19 smithi175 ceph-mon[36128]: pgmap v480: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:41:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:41:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:41:20.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:41:20.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 919688 12.07 MB/sec execute 296 sec latency 350.346 ms 2024-09-13T15:41:21.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 310013 4.01 MB/sec execute 297 sec latency 10.848 ms 2024-09-13T15:41:21.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 921887 12.07 MB/sec execute 297 sec latency 132.052 ms 2024-09-13T15:41:22.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:21 smithi090 ceph-mon[27811]: pgmap v481: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 MiB/s wr, 71 op/s 2024-09-13T15:41:22.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 310921 4.00 MB/sec execute 298 sec latency 10.084 ms 2024-09-13T15:41:22.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:21 smithi175 ceph-mon[36128]: pgmap v481: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 MiB/s wr, 71 op/s 2024-09-13T15:41:22.210 INFO:tasks.workunit.client.0.smithi090.stdout: 1 924779 12.08 MB/sec execute 298 sec latency 39.445 ms 2024-09-13T15:41:23.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 311904 4.01 MB/sec execute 299 sec latency 19.735 ms 2024-09-13T15:41:23.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 928142 12.10 MB/sec execute 299 sec latency 141.888 ms 2024-09-13T15:41:24.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:23 smithi090 ceph-mon[27811]: pgmap v482: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-09-13T15:41:24.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 312526 4.01 MB/sec execute 300 sec latency 12.912 ms 2024-09-13T15:41:24.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:23 smithi175 ceph-mon[36128]: pgmap v482: 129 pgs: 129 active+clean; 135 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2024-09-13T15:41:24.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 931623 12.12 MB/sec execute 300 sec latency 33.401 ms 2024-09-13T15:41:25.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 313185 4.01 MB/sec execute 301 sec latency 10.854 ms 2024-09-13T15:41:25.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 935117 12.15 MB/sec execute 301 sec latency 83.558 ms 2024-09-13T15:41:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:25 smithi090 ceph-mon[27811]: pgmap v483: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-13T15:41:26.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 314030 4.00 MB/sec execute 302 sec latency 13.819 ms 2024-09-13T15:41:26.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:25 smithi175 ceph-mon[36128]: pgmap v483: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-13T15:41:26.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 938433 12.17 MB/sec execute 302 sec latency 66.958 ms 2024-09-13T15:41:27.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 314907 4.01 MB/sec execute 303 sec latency 11.191 ms 2024-09-13T15:41:27.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 941308 12.18 MB/sec execute 303 sec latency 30.919 ms 2024-09-13T15:41:28.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:27 smithi090 ceph-mon[27811]: pgmap v484: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.6 MiB/s wr, 90 op/s 2024-09-13T15:41:28.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 315729 4.02 MB/sec execute 304 sec latency 18.769 ms 2024-09-13T15:41:28.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:27 smithi175 ceph-mon[36128]: pgmap v484: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.6 MiB/s wr, 90 op/s 2024-09-13T15:41:28.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 943940 12.18 MB/sec execute 304 sec latency 126.479 ms 2024-09-13T15:41:29.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 316324 4.01 MB/sec execute 305 sec latency 11.463 ms 2024-09-13T15:41:29.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 946995 12.19 MB/sec execute 305 sec latency 33.521 ms 2024-09-13T15:41:30.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:29 smithi090 ceph-mon[27811]: pgmap v485: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-13T15:41:30.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 317190 4.01 MB/sec execute 306 sec latency 12.961 ms 2024-09-13T15:41:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:29 smithi175 ceph-mon[36128]: pgmap v485: 129 pgs: 129 active+clean; 137 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-13T15:41:30.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 949645 12.20 MB/sec execute 306 sec latency 33.339 ms 2024-09-13T15:41:31.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 318062 4.01 MB/sec execute 307 sec latency 14.933 ms 2024-09-13T15:41:31.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 953008 12.22 MB/sec execute 307 sec latency 57.046 ms 2024-09-13T15:41:32.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 319006 4.02 MB/sec execute 308 sec latency 20.453 ms 2024-09-13T15:41:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:31 smithi175 ceph-mon[36128]: pgmap v486: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-09-13T15:41:32.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 956483 12.24 MB/sec execute 308 sec latency 134.247 ms 2024-09-13T15:41:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:31 smithi090 ceph-mon[27811]: pgmap v486: 129 pgs: 129 active+clean; 138 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-09-13T15:41:33.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 319628 4.02 MB/sec execute 309 sec latency 11.693 ms 2024-09-13T15:41:33.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 959962 12.27 MB/sec execute 309 sec latency 132.027 ms 2024-09-13T15:41:34.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 320315 4.01 MB/sec execute 310 sec latency 10.058 ms 2024-09-13T15:41:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:33 smithi175 ceph-mon[36128]: pgmap v487: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:41:34.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:34.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 962613 12.27 MB/sec execute 310 sec latency 148.542 ms 2024-09-13T15:41:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:33 smithi090 ceph-mon[27811]: pgmap v487: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:41:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:35.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 321224 4.01 MB/sec execute 311 sec latency 9.712 ms 2024-09-13T15:41:35.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 966028 12.29 MB/sec execute 311 sec latency 26.153 ms 2024-09-13T15:41:36.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 322013 4.02 MB/sec execute 312 sec latency 11.724 ms 2024-09-13T15:41:36.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:35 smithi175 ceph-mon[36128]: pgmap v488: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 105 op/s 2024-09-13T15:41:36.072 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:35 smithi090 ceph-mon[27811]: pgmap v488: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 105 op/s 2024-09-13T15:41:36.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 968857 12.29 MB/sec execute 312 sec latency 33.497 ms 2024-09-13T15:41:37.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 322899 4.03 MB/sec execute 313 sec latency 11.394 ms 2024-09-13T15:41:37.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 971818 12.31 MB/sec execute 313 sec latency 72.114 ms 2024-09-13T15:41:38.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 323462 4.02 MB/sec execute 314 sec latency 13.184 ms 2024-09-13T15:41:38.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:37 smithi175 ceph-mon[36128]: pgmap v489: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-09-13T15:41:38.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 974956 12.33 MB/sec execute 314 sec latency 33.357 ms 2024-09-13T15:41:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:37 smithi090 ceph-mon[27811]: pgmap v489: 129 pgs: 129 active+clean; 140 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-09-13T15:41:39.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 324306 4.02 MB/sec execute 315 sec latency 11.441 ms 2024-09-13T15:41:39.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 977829 12.34 MB/sec execute 315 sec latency 113.698 ms 2024-09-13T15:41:40.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:39 smithi175 ceph-mon[36128]: pgmap v490: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 96 op/s 2024-09-13T15:41:40.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 325158 4.01 MB/sec execute 316 sec latency 13.294 ms 2024-09-13T15:41:40.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 981360 12.36 MB/sec execute 316 sec latency 32.816 ms 2024-09-13T15:41:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:39 smithi090 ceph-mon[27811]: pgmap v490: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 96 op/s 2024-09-13T15:41:41.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 326090 4.02 MB/sec execute 317 sec latency 12.107 ms 2024-09-13T15:41:41.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 984792 12.38 MB/sec execute 317 sec latency 133.186 ms 2024-09-13T15:41:42.016 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:41 smithi175 ceph-mon[36128]: pgmap v491: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-13T15:41:42.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 326707 4.03 MB/sec execute 318 sec latency 11.645 ms 2024-09-13T15:41:42.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 987409 12.38 MB/sec execute 318 sec latency 114.385 ms 2024-09-13T15:41:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:41 smithi090 ceph-mon[27811]: pgmap v491: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-13T15:41:43.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 327380 4.02 MB/sec execute 319 sec latency 13.312 ms 2024-09-13T15:41:43.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 990450 12.39 MB/sec execute 319 sec latency 61.696 ms 2024-09-13T15:41:44.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:43 smithi175 ceph-mon[36128]: pgmap v492: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 93 op/s 2024-09-13T15:41:44.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 328213 4.02 MB/sec execute 320 sec latency 12.694 ms 2024-09-13T15:41:44.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 993649 12.40 MB/sec execute 320 sec latency 33.429 ms 2024-09-13T15:41:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:43 smithi090 ceph-mon[27811]: pgmap v492: 129 pgs: 129 active+clean; 142 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 93 op/s 2024-09-13T15:41:45.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 329097 4.02 MB/sec execute 321 sec latency 15.263 ms 2024-09-13T15:41:45.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 996665 12.42 MB/sec execute 321 sec latency 33.534 ms 2024-09-13T15:41:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:45 smithi175 ceph-mon[36128]: pgmap v493: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-09-13T15:41:46.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 329672 4.02 MB/sec execute 322 sec latency 123.217 ms 2024-09-13T15:41:46.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 999009 12.42 MB/sec execute 322 sec latency 466.679 ms 2024-09-13T15:41:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:45 smithi090 ceph-mon[27811]: pgmap v493: 129 pgs: 129 active+clean; 143 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 99 op/s 2024-09-13T15:41:47.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 330354 4.02 MB/sec execute 323 sec latency 11.653 ms 2024-09-13T15:41:47.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1001617 12.43 MB/sec execute 323 sec latency 117.214 ms 2024-09-13T15:41:48.015 INFO:tasks.workunit.client.1.smithi175.stdout: 1 331092 4.02 MB/sec execute 324 sec latency 11.217 ms 2024-09-13T15:41:48.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1004343 12.43 MB/sec execute 324 sec latency 64.720 ms 2024-09-13T15:41:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:47 smithi090 ceph-mon[27811]: pgmap v494: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:41:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:47 smithi175 ceph-mon[36128]: pgmap v494: 129 pgs: 129 active+clean; 141 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:41:49.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 331981 4.02 MB/sec execute 325 sec latency 11.623 ms 2024-09-13T15:41:49.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:49.211 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1006902 12.44 MB/sec execute 325 sec latency 125.053 ms 2024-09-13T15:41:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:41:50.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 332895 4.03 MB/sec execute 326 sec latency 17.358 ms 2024-09-13T15:41:50.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1009748 12.45 MB/sec execute 326 sec latency 143.637 ms 2024-09-13T15:41:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:49 smithi090 ceph-mon[27811]: pgmap v495: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-13T15:41:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:49 smithi175 ceph-mon[36128]: pgmap v495: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.0 MiB/s wr, 99 op/s 2024-09-13T15:41:51.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 333672 4.03 MB/sec execute 327 sec latency 15.749 ms 2024-09-13T15:41:51.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1011937 12.45 MB/sec execute 327 sec latency 210.669 ms 2024-09-13T15:41:52.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 333989 4.02 MB/sec execute 328 sec latency 129.722 ms 2024-09-13T15:41:52.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1012252 12.41 MB/sec execute 328 sec latency 393.511 ms 2024-09-13T15:41:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:51 smithi090 ceph-mon[27811]: pgmap v496: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:41:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:51 smithi175 ceph-mon[36128]: pgmap v496: 129 pgs: 129 active+clean; 144 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:41:53.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 334522 4.01 MB/sec execute 329 sec latency 143.821 ms 2024-09-13T15:41:53.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1012284 12.38 MB/sec execute 329 sec latency 380.347 ms 2024-09-13T15:41:54.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 335275 4.01 MB/sec execute 330 sec latency 80.903 ms 2024-09-13T15:41:54.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013183 12.35 MB/sec execute 330 sec latency 441.264 ms 2024-09-13T15:41:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:53 smithi090 ceph-mon[27811]: pgmap v497: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-13T15:41:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:53 smithi175 ceph-mon[36128]: pgmap v497: 129 pgs: 129 active+clean; 145 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-13T15:41:55.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 335976 4.01 MB/sec execute 331 sec latency 124.893 ms 2024-09-13T15:41:55.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013190 12.32 MB/sec execute 331 sec latency 413.209 ms 2024-09-13T15:41:56.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 336751 4.02 MB/sec execute 332 sec latency 73.047 ms 2024-09-13T15:41:56.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013239 12.28 MB/sec execute 332 sec latency 358.222 ms 2024-09-13T15:41:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:55 smithi090 ceph-mon[27811]: pgmap v498: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-09-13T15:41:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:55 smithi175 ceph-mon[36128]: pgmap v498: 129 pgs: 129 active+clean; 146 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-09-13T15:41:57.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 337221 4.02 MB/sec execute 333 sec latency 252.151 ms 2024-09-13T15:41:57.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013244 12.24 MB/sec execute 333 sec latency 481.407 ms 2024-09-13T15:41:58.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 337601 4.01 MB/sec execute 334 sec latency 84.427 ms 2024-09-13T15:41:58.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013305 12.21 MB/sec execute 334 sec latency 401.570 ms 2024-09-13T15:41:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:57 smithi090 ceph-mon[27811]: pgmap v499: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:41:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:57 smithi175 ceph-mon[36128]: pgmap v499: 129 pgs: 129 active+clean; 145 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:41:59.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 338147 4.00 MB/sec execute 335 sec latency 220.509 ms 2024-09-13T15:41:59.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013310 12.17 MB/sec execute 335 sec latency 528.590 ms 2024-09-13T15:42:00.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 338646 4.00 MB/sec execute 336 sec latency 155.078 ms 2024-09-13T15:42:00.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013317 12.13 MB/sec execute 336 sec latency 420.108 ms 2024-09-13T15:42:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:41:59 smithi090 ceph-mon[27811]: pgmap v500: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-09-13T15:42:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:41:59 smithi175 ceph-mon[36128]: pgmap v500: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-09-13T15:42:01.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 339340 3.99 MB/sec execute 337 sec latency 164.781 ms 2024-09-13T15:42:01.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013391 12.10 MB/sec execute 337 sec latency 397.547 ms 2024-09-13T15:42:02.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:01 smithi090 ceph-mon[27811]: pgmap v501: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-09-13T15:42:02.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 339948 4.00 MB/sec execute 338 sec latency 218.681 ms 2024-09-13T15:42:02.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013494 12.06 MB/sec execute 338 sec latency 509.889 ms 2024-09-13T15:42:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:01 smithi175 ceph-mon[36128]: pgmap v501: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.4 MiB/s wr, 71 op/s 2024-09-13T15:42:03.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 340641 4.00 MB/sec execute 339 sec latency 107.544 ms 2024-09-13T15:42:03.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1013992 12.04 MB/sec execute 339 sec latency 513.388 ms 2024-09-13T15:42:04.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 341082 3.99 MB/sec execute 340 sec latency 78.553 ms 2024-09-13T15:42:04.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1014394 12.01 MB/sec execute 340 sec latency 336.654 ms 2024-09-13T15:42:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:03 smithi090 ceph-mon[27811]: pgmap v502: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:42:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:03 smithi175 ceph-mon[36128]: pgmap v502: 129 pgs: 129 active+clean; 146 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 74 op/s 2024-09-13T15:42:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:05.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 341725 3.99 MB/sec execute 341 sec latency 66.100 ms 2024-09-13T15:42:05.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1014882 11.98 MB/sec execute 341 sec latency 351.378 ms 2024-09-13T15:42:06.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:05 smithi175 ceph-mon[36128]: pgmap v503: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:42:06.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 342627 3.98 MB/sec execute 342 sec latency 9.585 ms 2024-09-13T15:42:06.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1017039 11.98 MB/sec execute 342 sec latency 166.645 ms 2024-09-13T15:42:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:05 smithi090 ceph-mon[27811]: pgmap v503: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:42:07.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 343548 3.99 MB/sec execute 343 sec latency 11.353 ms 2024-09-13T15:42:07.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1020339 12.00 MB/sec execute 343 sec latency 141.516 ms 2024-09-13T15:42:08.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 344329 4.00 MB/sec execute 344 sec latency 13.779 ms 2024-09-13T15:42:08.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1023820 12.02 MB/sec execute 344 sec latency 75.134 ms 2024-09-13T15:42:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:07 smithi090 ceph-mon[27811]: pgmap v504: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 MiB/s wr, 68 op/s 2024-09-13T15:42:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:07 smithi175 ceph-mon[36128]: pgmap v504: 129 pgs: 129 active+clean; 147 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.4 MiB/s wr, 68 op/s 2024-09-13T15:42:09.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 344954 3.99 MB/sec execute 345 sec latency 10.832 ms 2024-09-13T15:42:09.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1026443 12.02 MB/sec execute 345 sec latency 33.471 ms 2024-09-13T15:42:10.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 345800 3.99 MB/sec execute 346 sec latency 15.545 ms 2024-09-13T15:42:10.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1028578 12.02 MB/sec execute 346 sec latency 244.676 ms 2024-09-13T15:42:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:09 smithi090 ceph-mon[27811]: pgmap v505: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-13T15:42:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:09 smithi175 ceph-mon[36128]: pgmap v505: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.7 MiB/s wr, 93 op/s 2024-09-13T15:42:11.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 346508 3.99 MB/sec execute 347 sec latency 133.046 ms 2024-09-13T15:42:11.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1028582 11.98 MB/sec execute 347 sec latency 417.404 ms 2024-09-13T15:42:12.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 347357 4.00 MB/sec execute 348 sec latency 87.953 ms 2024-09-13T15:42:12.212 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1029105 11.95 MB/sec execute 348 sec latency 393.356 ms 2024-09-13T15:42:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:11 smithi090 ceph-mon[27811]: pgmap v506: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-09-13T15:42:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:11 smithi175 ceph-mon[36128]: pgmap v506: 129 pgs: 129 active+clean; 148 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.3 MiB/s wr, 69 op/s 2024-09-13T15:42:13.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 347948 4.00 MB/sec execute 349 sec latency 68.030 ms 2024-09-13T15:42:13.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1031037 11.96 MB/sec execute 349 sec latency 301.861 ms 2024-09-13T15:42:14.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 348608 3.99 MB/sec execute 350 sec latency 10.942 ms 2024-09-13T15:42:14.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1034519 11.98 MB/sec execute 350 sec latency 33.402 ms 2024-09-13T15:42:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:13 smithi090 ceph-mon[27811]: pgmap v507: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:42:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:13 smithi175 ceph-mon[36128]: pgmap v507: 129 pgs: 129 active+clean; 149 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:42:15.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 349483 3.99 MB/sec execute 351 sec latency 13.217 ms 2024-09-13T15:42:15.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1038054 12.00 MB/sec execute 351 sec latency 33.334 ms 2024-09-13T15:42:16.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 350358 3.99 MB/sec execute 352 sec latency 11.781 ms 2024-09-13T15:42:16.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1040657 12.00 MB/sec execute 352 sec latency 65.574 ms 2024-09-13T15:42:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:15 smithi090 ceph-mon[27811]: pgmap v508: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-09-13T15:42:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:15 smithi175 ceph-mon[36128]: pgmap v508: 129 pgs: 129 active+clean; 150 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 2.0 MiB/s wr, 102 op/s 2024-09-13T15:42:17.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 351241 4.00 MB/sec execute 353 sec latency 10.373 ms 2024-09-13T15:42:17.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1043640 12.01 MB/sec execute 353 sec latency 95.760 ms 2024-09-13T15:42:18.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 351728 4.00 MB/sec execute 354 sec latency 128.261 ms 2024-09-13T15:42:18.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1045154 12.00 MB/sec execute 354 sec latency 351.948 ms 2024-09-13T15:42:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:17 smithi090 ceph-mon[27811]: pgmap v509: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:42:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:17 smithi175 ceph-mon[36128]: pgmap v509: 129 pgs: 129 active+clean; 150 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:42:19.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 352279 3.99 MB/sec execute 355 sec latency 82.840 ms 2024-09-13T15:42:19.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:19.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1045159 11.97 MB/sec execute 355 sec latency 424.836 ms 2024-09-13T15:42:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:20.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 352986 3.99 MB/sec execute 356 sec latency 113.545 ms 2024-09-13T15:42:20.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1045903 11.95 MB/sec execute 356 sec latency 368.205 ms 2024-09-13T15:42:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: pgmap v510: 129 pgs: 129 active+clean; 151 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 110 op/s 2024-09-13T15:42:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:42:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:42:20.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:42:20.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:42:20.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:42:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: pgmap v510: 129 pgs: 129 active+clean; 151 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 110 op/s 2024-09-13T15:42:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:42:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-13T15:42:20.267 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:42:20.267 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:42:20.267 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:42:20.267 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:42:21.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 353762 3.98 MB/sec execute 357 sec latency 12.246 ms 2024-09-13T15:42:21.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1048891 11.96 MB/sec execute 357 sec latency 33.741 ms 2024-09-13T15:42:21.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:20 smithi090 ceph-mon[27811]: Adjusting osd_memory_target on smithi175 to 1680M 2024-09-13T15:42:21.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:20 smithi175 ceph-mon[36128]: Adjusting osd_memory_target on smithi175 to 1680M 2024-09-13T15:42:22.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 354708 4.00 MB/sec execute 358 sec latency 11.716 ms 2024-09-13T15:42:22.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1052314 11.98 MB/sec execute 358 sec latency 74.792 ms 2024-09-13T15:42:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:21 smithi090 ceph-mon[27811]: pgmap v511: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:42:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:21 smithi175 ceph-mon[36128]: pgmap v511: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:42:23.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 355279 3.99 MB/sec execute 359 sec latency 11.258 ms 2024-09-13T15:42:23.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1055793 12.00 MB/sec execute 359 sec latency 33.516 ms 2024-09-13T15:42:24.016 INFO:tasks.workunit.client.1.smithi175.stdout: 1 356124 4.00 MB/sec execute 360 sec latency 12.083 ms 2024-09-13T15:42:24.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1059275 12.02 MB/sec execute 360 sec latency 93.506 ms 2024-09-13T15:42:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:23 smithi090 ceph-mon[27811]: pgmap v512: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-13T15:42:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:23 smithi175 ceph-mon[36128]: pgmap v512: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-13T15:42:25.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 357000 3.99 MB/sec execute 361 sec latency 28.459 ms 2024-09-13T15:42:25.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1060087 12.00 MB/sec execute 361 sec latency 282.653 ms 2024-09-13T15:42:26.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 357873 4.00 MB/sec execute 362 sec latency 77.817 ms 2024-09-13T15:42:26.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1060489 11.96 MB/sec execute 362 sec latency 371.972 ms 2024-09-13T15:42:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:25 smithi090 ceph-mon[27811]: pgmap v513: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 105 op/s 2024-09-13T15:42:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:25 smithi175 ceph-mon[36128]: pgmap v513: 129 pgs: 129 active+clean; 153 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 105 op/s 2024-09-13T15:42:27.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 358447 4.00 MB/sec execute 363 sec latency 147.816 ms 2024-09-13T15:42:27.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1061924 11.96 MB/sec execute 363 sec latency 385.649 ms 2024-09-13T15:42:28.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 358973 3.99 MB/sec execute 364 sec latency 13.000 ms 2024-09-13T15:42:28.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1064566 11.95 MB/sec execute 364 sec latency 96.702 ms 2024-09-13T15:42:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:27 smithi090 ceph-mon[27811]: pgmap v514: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:42:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:27 smithi175 ceph-mon[36128]: pgmap v514: 129 pgs: 129 active+clean; 152 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 83 op/s 2024-09-13T15:42:29.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 359839 3.99 MB/sec execute 365 sec latency 14.862 ms 2024-09-13T15:42:29.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1067580 11.97 MB/sec execute 365 sec latency 33.305 ms 2024-09-13T15:42:30.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 360683 3.99 MB/sec execute 366 sec latency 10.419 ms 2024-09-13T15:42:30.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1070125 11.98 MB/sec execute 366 sec latency 141.815 ms 2024-09-13T15:42:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:29 smithi090 ceph-mon[27811]: pgmap v515: 129 pgs: 129 active+clean; 154 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-09-13T15:42:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:29 smithi175 ceph-mon[36128]: pgmap v515: 129 pgs: 129 active+clean; 154 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-09-13T15:42:31.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 361572 4.00 MB/sec execute 367 sec latency 18.144 ms 2024-09-13T15:42:31.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1073513 11.99 MB/sec execute 367 sec latency 141.792 ms 2024-09-13T15:42:32.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 362240 4.00 MB/sec execute 368 sec latency 11.527 ms 2024-09-13T15:42:32.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1075547 11.99 MB/sec execute 368 sec latency 251.992 ms 2024-09-13T15:42:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:31 smithi090 ceph-mon[27811]: pgmap v516: 129 pgs: 129 active+clean; 154 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-09-13T15:42:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:31 smithi175 ceph-mon[36128]: pgmap v516: 129 pgs: 129 active+clean; 154 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 75 op/s 2024-09-13T15:42:33.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 362765 3.99 MB/sec execute 369 sec latency 111.199 ms 2024-09-13T15:42:33.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1075760 11.96 MB/sec execute 369 sec latency 519.210 ms 2024-09-13T15:42:34.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 363500 3.99 MB/sec execute 370 sec latency 86.297 ms 2024-09-13T15:42:34.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1076075 11.94 MB/sec execute 370 sec latency 371.861 ms 2024-09-13T15:42:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:33 smithi090 ceph-mon[27811]: pgmap v517: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-09-13T15:42:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:34.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:33 smithi175 ceph-mon[36128]: pgmap v517: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.9 MiB/s wr, 104 op/s 2024-09-13T15:42:34.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:35.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 364226 3.99 MB/sec execute 371 sec latency 96.491 ms 2024-09-13T15:42:35.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1077133 11.92 MB/sec execute 371 sec latency 475.012 ms 2024-09-13T15:42:36.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 365115 3.99 MB/sec execute 372 sec latency 16.284 ms 2024-09-13T15:42:36.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1079631 11.92 MB/sec execute 372 sec latency 223.327 ms 2024-09-13T15:42:36.241 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:35 smithi175 ceph-mon[36128]: pgmap v518: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:42:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:35 smithi090 ceph-mon[27811]: pgmap v518: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 100 op/s 2024-09-13T15:42:37.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 365786 4.00 MB/sec execute 373 sec latency 11.623 ms 2024-09-13T15:42:37.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1081765 11.92 MB/sec execute 373 sec latency 166.854 ms 2024-09-13T15:42:38.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 366532 3.99 MB/sec execute 374 sec latency 8.784 ms 2024-09-13T15:42:38.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1084223 11.92 MB/sec execute 374 sec latency 174.932 ms 2024-09-13T15:42:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:37 smithi090 ceph-mon[27811]: pgmap v519: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:42:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:37 smithi175 ceph-mon[36128]: pgmap v519: 129 pgs: 129 active+clean; 156 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:42:39.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 367422 3.99 MB/sec execute 375 sec latency 13.438 ms 2024-09-13T15:42:39.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1087232 11.94 MB/sec execute 375 sec latency 141.256 ms 2024-09-13T15:42:40.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 368238 4.00 MB/sec execute 376 sec latency 76.857 ms 2024-09-13T15:42:40.213 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1087768 11.91 MB/sec execute 376 sec latency 343.494 ms 2024-09-13T15:42:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:39 smithi090 ceph-mon[27811]: pgmap v520: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-09-13T15:42:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:39 smithi175 ceph-mon[36128]: pgmap v520: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.0 MiB/s wr, 107 op/s 2024-09-13T15:42:41.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 368716 3.99 MB/sec execute 377 sec latency 122.833 ms 2024-09-13T15:42:41.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1087773 11.88 MB/sec execute 377 sec latency 361.200 ms 2024-09-13T15:42:42.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 369274 3.99 MB/sec execute 378 sec latency 198.118 ms 2024-09-13T15:42:42.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1089872 11.88 MB/sec execute 378 sec latency 411.267 ms 2024-09-13T15:42:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:41 smithi090 ceph-mon[27811]: pgmap v521: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:42:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:41 smithi175 ceph-mon[36128]: pgmap v521: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:42:43.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 369956 3.99 MB/sec execute 379 sec latency 11.707 ms 2024-09-13T15:42:43.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1092856 11.88 MB/sec execute 379 sec latency 46.017 ms 2024-09-13T15:42:44.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 370840 3.99 MB/sec execute 380 sec latency 10.190 ms 2024-09-13T15:42:44.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1094980 11.89 MB/sec execute 380 sec latency 108.625 ms 2024-09-13T15:42:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:43 smithi090 ceph-mon[27811]: pgmap v522: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-13T15:42:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:43 smithi175 ceph-mon[36128]: pgmap v522: 129 pgs: 129 active+clean; 157 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-13T15:42:45.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 371759 3.99 MB/sec execute 381 sec latency 17.752 ms 2024-09-13T15:42:45.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1098340 11.91 MB/sec execute 381 sec latency 134.847 ms 2024-09-13T15:42:46.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 372565 4.00 MB/sec execute 382 sec latency 12.094 ms 2024-09-13T15:42:46.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1101832 11.92 MB/sec execute 382 sec latency 33.256 ms 2024-09-13T15:42:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:45 smithi090 ceph-mon[27811]: pgmap v523: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-13T15:42:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:45 smithi175 ceph-mon[36128]: pgmap v523: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.8 MiB/s wr, 82 op/s 2024-09-13T15:42:47.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 373155 3.99 MB/sec execute 383 sec latency 9.334 ms 2024-09-13T15:42:47.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1103036 11.91 MB/sec execute 383 sec latency 280.747 ms 2024-09-13T15:42:48.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:47 smithi090 ceph-mon[27811]: pgmap v524: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-13T15:42:48.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 373886 4.00 MB/sec execute 384 sec latency 92.379 ms 2024-09-13T15:42:48.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1103521 11.88 MB/sec execute 384 sec latency 396.262 ms 2024-09-13T15:42:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:47 smithi175 ceph-mon[36128]: pgmap v524: 129 pgs: 129 active+clean; 158 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 80 op/s 2024-09-13T15:42:49.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 374487 3.99 MB/sec execute 385 sec latency 145.410 ms 2024-09-13T15:42:49.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:49.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1104123 11.86 MB/sec execute 385 sec latency 396.290 ms 2024-09-13T15:42:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:42:50.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 375418 3.99 MB/sec execute 386 sec latency 80.719 ms 2024-09-13T15:42:50.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1105686 11.85 MB/sec execute 386 sec latency 129.072 ms 2024-09-13T15:42:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:49 smithi090 ceph-mon[27811]: pgmap v525: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-09-13T15:42:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:49 smithi175 ceph-mon[36128]: pgmap v525: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.1 MiB/s wr, 103 op/s 2024-09-13T15:42:51.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 376210 4.00 MB/sec execute 387 sec latency 11.101 ms 2024-09-13T15:42:51.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1109122 11.87 MB/sec execute 387 sec latency 33.297 ms 2024-09-13T15:42:52.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 376764 3.99 MB/sec execute 388 sec latency 11.977 ms 2024-09-13T15:42:52.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1112585 11.89 MB/sec execute 388 sec latency 45.100 ms 2024-09-13T15:42:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:51 smithi090 ceph-mon[27811]: pgmap v526: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:42:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:51 smithi175 ceph-mon[36128]: pgmap v526: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:42:53.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 377590 3.99 MB/sec execute 389 sec latency 12.207 ms 2024-09-13T15:42:53.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1116064 11.91 MB/sec execute 389 sec latency 33.435 ms 2024-09-13T15:42:54.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 378446 3.99 MB/sec execute 390 sec latency 17.311 ms 2024-09-13T15:42:54.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1119557 11.92 MB/sec execute 390 sec latency 41.491 ms 2024-09-13T15:42:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:53 smithi090 ceph-mon[27811]: pgmap v527: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-13T15:42:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:53 smithi175 ceph-mon[36128]: pgmap v527: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-13T15:42:55.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 379306 4.00 MB/sec execute 391 sec latency 126.538 ms 2024-09-13T15:42:55.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1119679 11.89 MB/sec execute 391 sec latency 370.299 ms 2024-09-13T15:42:56.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 379764 4.00 MB/sec execute 392 sec latency 153.621 ms 2024-09-13T15:42:56.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1119686 11.86 MB/sec execute 392 sec latency 384.088 ms 2024-09-13T15:42:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:55 smithi090 ceph-mon[27811]: pgmap v528: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 105 op/s 2024-09-13T15:42:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:55 smithi175 ceph-mon[36128]: pgmap v528: 129 pgs: 129 active+clean; 161 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 105 op/s 2024-09-13T15:42:57.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 380176 3.99 MB/sec execute 393 sec latency 127.780 ms 2024-09-13T15:42:57.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1122177 11.87 MB/sec execute 393 sec latency 356.443 ms 2024-09-13T15:42:58.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 381018 3.99 MB/sec execute 394 sec latency 11.224 ms 2024-09-13T15:42:58.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1125335 11.88 MB/sec execute 394 sec latency 39.458 ms 2024-09-13T15:42:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:57 smithi090 ceph-mon[27811]: pgmap v529: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:42:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:57 smithi175 ceph-mon[36128]: pgmap v529: 129 pgs: 129 active+clean; 160 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:42:59.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 381898 3.99 MB/sec execute 395 sec latency 14.985 ms 2024-09-13T15:42:59.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1127859 11.88 MB/sec execute 395 sec latency 99.167 ms 2024-09-13T15:43:00.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 382862 3.99 MB/sec execute 396 sec latency 12.082 ms 2024-09-13T15:43:00.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1130499 11.89 MB/sec execute 396 sec latency 33.536 ms 2024-09-13T15:43:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:42:59 smithi090 ceph-mon[27811]: pgmap v530: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-09-13T15:43:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:42:59 smithi175 ceph-mon[36128]: pgmap v530: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 103 op/s 2024-09-13T15:43:01.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 383487 4.00 MB/sec execute 397 sec latency 9.378 ms 2024-09-13T15:43:01.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1133918 11.91 MB/sec execute 397 sec latency 57.686 ms 2024-09-13T15:43:02.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:01 smithi175 ceph-mon[36128]: pgmap v531: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:43:02.017 INFO:tasks.workunit.client.1.smithi175.stdout: 1 384107 3.99 MB/sec execute 398 sec latency 84.603 ms 2024-09-13T15:43:02.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1135877 11.90 MB/sec execute 398 sec latency 332.373 ms 2024-09-13T15:43:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:01 smithi090 ceph-mon[27811]: pgmap v531: 129 pgs: 129 active+clean; 162 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 75 op/s 2024-09-13T15:43:03.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 384751 3.99 MB/sec execute 399 sec latency 127.963 ms 2024-09-13T15:43:03.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1136108 11.88 MB/sec execute 399 sec latency 372.250 ms 2024-09-13T15:43:04.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 385492 3.98 MB/sec execute 400 sec latency 130.572 ms 2024-09-13T15:43:04.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1137393 11.87 MB/sec execute 400 sec latency 424.913 ms 2024-09-13T15:43:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:03 smithi090 ceph-mon[27811]: pgmap v532: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:43:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:03 smithi175 ceph-mon[36128]: pgmap v532: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:43:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:05.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 386433 3.99 MB/sec execute 401 sec latency 11.523 ms 2024-09-13T15:43:05.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1140771 11.88 MB/sec execute 401 sec latency 143.352 ms 2024-09-13T15:43:06.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 387061 3.99 MB/sec execute 402 sec latency 11.893 ms 2024-09-13T15:43:06.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1143396 11.89 MB/sec execute 402 sec latency 33.099 ms 2024-09-13T15:43:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:05 smithi090 ceph-mon[27811]: pgmap v533: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 102 op/s 2024-09-13T15:43:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:05 smithi175 ceph-mon[36128]: pgmap v533: 129 pgs: 129 active+clean; 163 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.9 MiB/s wr, 102 op/s 2024-09-13T15:43:07.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 387749 3.99 MB/sec execute 403 sec latency 10.918 ms 2024-09-13T15:43:07.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1145638 11.88 MB/sec execute 403 sec latency 90.413 ms 2024-09-13T15:43:08.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 388636 3.99 MB/sec execute 404 sec latency 18.113 ms 2024-09-13T15:43:08.143 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:07 smithi090 ceph-mon[27811]: pgmap v534: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-13T15:43:08.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1148930 11.90 MB/sec execute 404 sec latency 33.550 ms 2024-09-13T15:43:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:07 smithi175 ceph-mon[36128]: pgmap v534: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-13T15:43:09.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 389571 3.99 MB/sec execute 405 sec latency 11.157 ms 2024-09-13T15:43:09.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1151568 11.91 MB/sec execute 405 sec latency 272.972 ms 2024-09-13T15:43:10.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 390200 4.00 MB/sec execute 406 sec latency 108.321 ms 2024-09-13T15:43:10.214 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1151575 11.88 MB/sec execute 406 sec latency 369.499 ms 2024-09-13T15:43:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:09 smithi090 ceph-mon[27811]: pgmap v535: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-13T15:43:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:09 smithi175 ceph-mon[36128]: pgmap v535: 129 pgs: 129 active+clean; 164 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 95 op/s 2024-09-13T15:43:11.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 390684 3.99 MB/sec execute 407 sec latency 95.460 ms 2024-09-13T15:43:11.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1151719 11.85 MB/sec execute 407 sec latency 466.433 ms 2024-09-13T15:43:12.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 391405 3.99 MB/sec execute 408 sec latency 93.282 ms 2024-09-13T15:43:12.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1155122 11.87 MB/sec execute 408 sec latency 126.573 ms 2024-09-13T15:43:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:11 smithi090 ceph-mon[27811]: pgmap v536: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:43:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:11 smithi175 ceph-mon[36128]: pgmap v536: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.6 MiB/s wr, 80 op/s 2024-09-13T15:43:13.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 392304 3.99 MB/sec execute 409 sec latency 17.116 ms 2024-09-13T15:43:13.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1158608 11.88 MB/sec execute 409 sec latency 64.603 ms 2024-09-13T15:43:14.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 393241 3.99 MB/sec execute 410 sec latency 14.417 ms 2024-09-13T15:43:14.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1161229 11.89 MB/sec execute 410 sec latency 33.961 ms 2024-09-13T15:43:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:13 smithi090 ceph-mon[27811]: pgmap v537: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-13T15:43:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:13 smithi175 ceph-mon[36128]: pgmap v537: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.7 MiB/s wr, 88 op/s 2024-09-13T15:43:15.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 394004 4.00 MB/sec execute 411 sec latency 11.265 ms 2024-09-13T15:43:15.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1164395 11.90 MB/sec execute 411 sec latency 41.839 ms 2024-09-13T15:43:16.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 394591 3.99 MB/sec execute 412 sec latency 11.563 ms 2024-09-13T15:43:16.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1166007 11.89 MB/sec execute 412 sec latency 142.992 ms 2024-09-13T15:43:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:15 smithi090 ceph-mon[27811]: pgmap v538: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.0 MiB/s wr, 109 op/s 2024-09-13T15:43:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:15 smithi175 ceph-mon[36128]: pgmap v538: 129 pgs: 129 active+clean; 166 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 2.0 MiB/s wr, 109 op/s 2024-09-13T15:43:17.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 395264 3.99 MB/sec execute 413 sec latency 127.557 ms 2024-09-13T15:43:17.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1166655 11.87 MB/sec execute 413 sec latency 372.107 ms 2024-09-13T15:43:18.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 395960 3.98 MB/sec execute 414 sec latency 142.703 ms 2024-09-13T15:43:18.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1167783 11.86 MB/sec execute 414 sec latency 351.575 ms 2024-09-13T15:43:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:17 smithi090 ceph-mon[27811]: pgmap v539: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:43:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:17 smithi175 ceph-mon[36128]: pgmap v539: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:43:19.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 397030 4.00 MB/sec execute 415 sec latency 12.130 ms 2024-09-13T15:43:19.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:19.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1169560 11.85 MB/sec execute 415 sec latency 127.949 ms 2024-09-13T15:43:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:20.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 397585 4.00 MB/sec execute 416 sec latency 11.286 ms 2024-09-13T15:43:20.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1172792 11.87 MB/sec execute 416 sec latency 233.664 ms 2024-09-13T15:43:20.217 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:19 smithi090 ceph-mon[27811]: pgmap v540: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-13T15:43:20.217 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:43:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:19 smithi175 ceph-mon[36128]: pgmap v540: 129 pgs: 129 active+clean; 165 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-13T15:43:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:43:21.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 398064 3.99 MB/sec execute 417 sec latency 89.033 ms 2024-09-13T15:43:21.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1174487 11.86 MB/sec execute 417 sec latency 65.995 ms 2024-09-13T15:43:22.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 398762 3.99 MB/sec execute 418 sec latency 113.999 ms 2024-09-13T15:43:22.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1176460 11.86 MB/sec execute 418 sec latency 73.822 ms 2024-09-13T15:43:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:21 smithi090 ceph-mon[27811]: pgmap v541: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:43:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:21 smithi175 ceph-mon[36128]: pgmap v541: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:43:23.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 399489 3.98 MB/sec execute 419 sec latency 81.906 ms 2024-09-13T15:43:23.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1178637 11.86 MB/sec execute 419 sec latency 66.176 ms 2024-09-13T15:43:24.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 400301 3.99 MB/sec execute 420 sec latency 114.365 ms 2024-09-13T15:43:24.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1180006 11.85 MB/sec execute 420 sec latency 68.246 ms 2024-09-13T15:43:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:23 smithi090 ceph-mon[27811]: pgmap v542: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:43:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:23 smithi175 ceph-mon[36128]: pgmap v542: 129 pgs: 129 active+clean; 167 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:43:25.018 INFO:tasks.workunit.client.1.smithi175.stdout: 1 401066 3.99 MB/sec execute 421 sec latency 13.344 ms 2024-09-13T15:43:25.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1182197 11.85 MB/sec execute 421 sec latency 66.937 ms 2024-09-13T15:43:26.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 401553 3.99 MB/sec execute 422 sec latency 12.833 ms 2024-09-13T15:43:26.215 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1184638 11.85 MB/sec execute 422 sec latency 52.320 ms 2024-09-13T15:43:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:25 smithi090 ceph-mon[27811]: pgmap v543: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-13T15:43:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:25 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:25 smithi175 ceph-mon[36128]: pgmap v543: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-09-13T15:43:26.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:26.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:25 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:27.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 402388 3.99 MB/sec execute 423 sec latency 16.686 ms 2024-09-13T15:43:27.152 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:43:27.152 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:43:27.152 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:27.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1187283 11.85 MB/sec execute 423 sec latency 38.653 ms 2024-09-13T15:43:27.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:43:27.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:43:27.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:43:28.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 403222 3.98 MB/sec execute 424 sec latency 15.232 ms 2024-09-13T15:43:28.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1190723 11.87 MB/sec execute 424 sec latency 45.054 ms 2024-09-13T15:43:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:27 smithi090 ceph-mon[27811]: pgmap v544: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-13T15:43:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:27 smithi175 ceph-mon[36128]: pgmap v544: 129 pgs: 129 active+clean; 168 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 76 op/s 2024-09-13T15:43:29.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 404165 3.99 MB/sec execute 425 sec latency 11.969 ms 2024-09-13T15:43:29.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1194184 11.89 MB/sec execute 425 sec latency 41.563 ms 2024-09-13T15:43:30.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 404803 3.99 MB/sec execute 426 sec latency 8.943 ms 2024-09-13T15:43:30.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1197668 11.90 MB/sec execute 426 sec latency 59.518 ms 2024-09-13T15:43:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:29 smithi090 ceph-mon[27811]: pgmap v545: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-09-13T15:43:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:29 smithi175 ceph-mon[36128]: pgmap v545: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 98 op/s 2024-09-13T15:43:31.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 405495 3.99 MB/sec execute 427 sec latency 10.589 ms 2024-09-13T15:43:31.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1201154 11.92 MB/sec execute 427 sec latency 50.173 ms 2024-09-13T15:43:32.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 406379 3.99 MB/sec execute 428 sec latency 12.062 ms 2024-09-13T15:43:32.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1203799 11.93 MB/sec execute 428 sec latency 32.997 ms 2024-09-13T15:43:32.230 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:31 smithi175 ceph-mon[36128]: pgmap v546: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-13T15:43:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:31 smithi090 ceph-mon[27811]: pgmap v546: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 89 op/s 2024-09-13T15:43:33.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 407301 3.99 MB/sec execute 429 sec latency 10.262 ms 2024-09-13T15:43:33.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1207018 11.94 MB/sec execute 429 sec latency 33.276 ms 2024-09-13T15:43:34.004 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:33 smithi090 ceph-mon[27811]: pgmap v547: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:43:34.004 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:34.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 408091 4.00 MB/sec execute 430 sec latency 12.166 ms 2024-09-13T15:43:34.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1209980 11.94 MB/sec execute 430 sec latency 41.942 ms 2024-09-13T15:43:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:33 smithi175 ceph-mon[36128]: pgmap v547: 129 pgs: 129 active+clean; 169 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-09-13T15:43:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:35.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 408659 3.99 MB/sec execute 431 sec latency 11.161 ms 2024-09-13T15:43:35.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1212751 11.96 MB/sec execute 431 sec latency 50.109 ms 2024-09-13T15:43:36.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 409509 3.99 MB/sec execute 432 sec latency 9.088 ms 2024-09-13T15:43:36.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1215469 11.96 MB/sec execute 432 sec latency 120.044 ms 2024-09-13T15:43:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:35 smithi090 ceph-mon[27811]: pgmap v548: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-09-13T15:43:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:35 smithi175 ceph-mon[36128]: pgmap v548: 129 pgs: 129 active+clean; 172 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-09-13T15:43:37.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 410350 3.99 MB/sec execute 433 sec latency 14.718 ms 2024-09-13T15:43:37.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1218950 11.98 MB/sec execute 433 sec latency 34.706 ms 2024-09-13T15:43:38.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 411267 4.00 MB/sec execute 434 sec latency 11.252 ms 2024-09-13T15:43:38.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1222437 11.99 MB/sec execute 434 sec latency 88.306 ms 2024-09-13T15:43:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:37 smithi090 ceph-mon[27811]: pgmap v549: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-13T15:43:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:37 smithi175 ceph-mon[36128]: pgmap v549: 129 pgs: 129 active+clean; 171 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 86 op/s 2024-09-13T15:43:39.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 411896 4.00 MB/sec execute 435 sec latency 45.459 ms 2024-09-13T15:43:39.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1223643 11.98 MB/sec execute 435 sec latency 306.147 ms 2024-09-13T15:43:40.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 412564 3.99 MB/sec execute 436 sec latency 12.507 ms 2024-09-13T15:43:40.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1226184 11.98 MB/sec execute 436 sec latency 129.057 ms 2024-09-13T15:43:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:39 smithi090 ceph-mon[27811]: pgmap v550: 129 pgs: 129 active+clean; 173 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 104 op/s 2024-09-13T15:43:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:39 smithi175 ceph-mon[36128]: pgmap v550: 129 pgs: 129 active+clean; 173 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.1 MiB/s wr, 104 op/s 2024-09-13T15:43:41.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 413461 3.99 MB/sec execute 437 sec latency 10.767 ms 2024-09-13T15:43:41.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1229525 12.00 MB/sec execute 437 sec latency 135.381 ms 2024-09-13T15:43:42.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 414366 4.00 MB/sec execute 438 sec latency 12.408 ms 2024-09-13T15:43:42.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1232146 12.00 MB/sec execute 438 sec latency 158.597 ms 2024-09-13T15:43:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:41 smithi090 ceph-mon[27811]: pgmap v551: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:43:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:41 smithi175 ceph-mon[36128]: pgmap v551: 129 pgs: 129 active+clean; 172 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.6 MiB/s wr, 77 op/s 2024-09-13T15:43:43.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 415145 4.00 MB/sec execute 439 sec latency 10.886 ms 2024-09-13T15:43:43.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1234032 12.00 MB/sec execute 439 sec latency 168.084 ms 2024-09-13T15:43:44.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 415722 4.00 MB/sec execute 440 sec latency 10.612 ms 2024-09-13T15:43:44.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1236818 12.00 MB/sec execute 440 sec latency 81.985 ms 2024-09-13T15:43:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:43 smithi090 ceph-mon[27811]: pgmap v552: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-13T15:43:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:43 smithi175 ceph-mon[36128]: pgmap v552: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-13T15:43:45.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 416423 4.00 MB/sec execute 441 sec latency 99.109 ms 2024-09-13T15:43:45.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1236861 11.98 MB/sec execute 441 sec latency 323.432 ms 2024-09-13T15:43:46.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 417096 3.99 MB/sec execute 442 sec latency 93.304 ms 2024-09-13T15:43:46.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1237422 11.96 MB/sec execute 442 sec latency 342.838 ms 2024-09-13T15:43:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:45 smithi090 ceph-mon[27811]: pgmap v553: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 105 op/s 2024-09-13T15:43:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:45 smithi175 ceph-mon[36128]: pgmap v553: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.2 MiB/s wr, 105 op/s 2024-09-13T15:43:47.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 417785 3.99 MB/sec execute 443 sec latency 136.749 ms 2024-09-13T15:43:47.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1237824 11.93 MB/sec execute 443 sec latency 351.023 ms 2024-09-13T15:43:48.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 418372 3.99 MB/sec execute 444 sec latency 139.710 ms 2024-09-13T15:43:48.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1238281 11.91 MB/sec execute 444 sec latency 340.129 ms 2024-09-13T15:43:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:47 smithi090 ceph-mon[27811]: pgmap v554: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-13T15:43:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:47 smithi175 ceph-mon[36128]: pgmap v554: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.9 MiB/s wr, 97 op/s 2024-09-13T15:43:49.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 418906 3.99 MB/sec execute 445 sec latency 143.547 ms 2024-09-13T15:43:49.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1238413 11.89 MB/sec execute 445 sec latency 366.455 ms 2024-09-13T15:43:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:43:50.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 419316 3.99 MB/sec execute 446 sec latency 139.596 ms 2024-09-13T15:43:50.178 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:49 smithi175 ceph-mon[36128]: pgmap v555: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-09-13T15:43:50.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1238914 11.87 MB/sec execute 446 sec latency 340.766 ms 2024-09-13T15:43:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:49 smithi090 ceph-mon[27811]: pgmap v555: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 100 op/s 2024-09-13T15:43:51.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 419870 3.98 MB/sec execute 447 sec latency 132.099 ms 2024-09-13T15:43:51.216 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1239229 11.85 MB/sec execute 447 sec latency 353.983 ms 2024-09-13T15:43:52.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 420428 3.98 MB/sec execute 448 sec latency 159.795 ms 2024-09-13T15:43:52.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1239271 11.82 MB/sec execute 448 sec latency 375.301 ms 2024-09-13T15:43:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:51 smithi090 ceph-mon[27811]: pgmap v556: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:43:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:51 smithi175 ceph-mon[36128]: pgmap v556: 129 pgs: 129 active+clean; 174 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 MiB/s wr, 77 op/s 2024-09-13T15:43:53.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 421057 3.98 MB/sec execute 449 sec latency 136.725 ms 2024-09-13T15:43:53.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1240164 11.81 MB/sec execute 449 sec latency 360.972 ms 2024-09-13T15:43:54.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 421879 3.98 MB/sec execute 450 sec latency 147.378 ms 2024-09-13T15:43:54.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1240227 11.78 MB/sec execute 450 sec latency 378.536 ms 2024-09-13T15:43:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:53 smithi090 ceph-mon[27811]: pgmap v557: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 100 op/s 2024-09-13T15:43:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:53 smithi175 ceph-mon[36128]: pgmap v557: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 100 op/s 2024-09-13T15:43:55.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 422509 3.98 MB/sec execute 451 sec latency 11.950 ms 2024-09-13T15:43:55.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1243702 11.80 MB/sec execute 451 sec latency 137.812 ms 2024-09-13T15:43:56.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 423204 3.98 MB/sec execute 452 sec latency 8.991 ms 2024-09-13T15:43:56.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1246220 11.80 MB/sec execute 452 sec latency 33.563 ms 2024-09-13T15:43:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:55 smithi090 ceph-mon[27811]: pgmap v558: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:43:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:55 smithi175 ceph-mon[36128]: pgmap v558: 129 pgs: 129 active+clean; 175 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.7 MiB/s wr, 81 op/s 2024-09-13T15:43:57.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 424090 3.98 MB/sec execute 453 sec latency 10.371 ms 2024-09-13T15:43:57.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1248991 11.80 MB/sec execute 453 sec latency 65.271 ms 2024-09-13T15:43:58.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:57 smithi175 ceph-mon[36128]: pgmap v559: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-13T15:43:58.019 INFO:tasks.workunit.client.1.smithi175.stdout: 1 425009 3.98 MB/sec execute 454 sec latency 10.592 ms 2024-09-13T15:43:58.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1251188 11.81 MB/sec execute 454 sec latency 61.797 ms 2024-09-13T15:43:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:57 smithi090 ceph-mon[27811]: pgmap v559: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-13T15:43:59.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 425763 3.99 MB/sec execute 455 sec latency 11.854 ms 2024-09-13T15:43:59.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1254551 11.82 MB/sec execute 455 sec latency 33.331 ms 2024-09-13T15:44:00.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 426340 3.98 MB/sec execute 456 sec latency 10.744 ms 2024-09-13T15:44:00.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1258009 11.83 MB/sec execute 456 sec latency 42.850 ms 2024-09-13T15:44:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:43:59 smithi090 ceph-mon[27811]: pgmap v560: 129 pgs: 129 active+clean; 177 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-13T15:44:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:43:59 smithi175 ceph-mon[36128]: pgmap v560: 129 pgs: 129 active+clean; 177 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.7 MiB/s wr, 85 op/s 2024-09-13T15:44:01.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 427203 3.98 MB/sec execute 457 sec latency 13.236 ms 2024-09-13T15:44:01.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1261495 11.85 MB/sec execute 457 sec latency 33.453 ms 2024-09-13T15:44:02.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 428040 3.98 MB/sec execute 458 sec latency 19.798 ms 2024-09-13T15:44:02.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1264987 11.87 MB/sec execute 458 sec latency 33.504 ms 2024-09-13T15:44:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:01 smithi090 ceph-mon[27811]: pgmap v561: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 MiB/s wr, 75 op/s 2024-09-13T15:44:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:01 smithi175 ceph-mon[36128]: pgmap v561: 129 pgs: 129 active+clean; 176 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.4 MiB/s wr, 75 op/s 2024-09-13T15:44:03.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 428985 3.99 MB/sec execute 459 sec latency 13.323 ms 2024-09-13T15:44:03.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1267492 11.87 MB/sec execute 459 sec latency 65.567 ms 2024-09-13T15:44:04.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 429632 3.99 MB/sec execute 460 sec latency 10.284 ms 2024-09-13T15:44:04.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1270613 11.88 MB/sec execute 460 sec latency 30.804 ms 2024-09-13T15:44:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:03 smithi090 ceph-mon[27811]: pgmap v562: 129 pgs: 129 active+clean; 177 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:44:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:04.268 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:03 smithi175 ceph-mon[36128]: pgmap v562: 129 pgs: 129 active+clean; 177 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:44:04.268 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:05.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 430344 3.99 MB/sec execute 461 sec latency 10.634 ms 2024-09-13T15:44:05.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1273280 11.88 MB/sec execute 461 sec latency 33.502 ms 2024-09-13T15:44:06.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 431248 3.98 MB/sec execute 462 sec latency 10.149 ms 2024-09-13T15:44:06.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1276425 11.90 MB/sec execute 462 sec latency 33.272 ms 2024-09-13T15:44:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:05 smithi090 ceph-mon[27811]: pgmap v563: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-13T15:44:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:05 smithi175 ceph-mon[36128]: pgmap v563: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 1.9 MiB/s wr, 90 op/s 2024-09-13T15:44:07.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 432188 3.99 MB/sec execute 463 sec latency 14.972 ms 2024-09-13T15:44:07.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1279363 11.90 MB/sec execute 463 sec latency 34.727 ms 2024-09-13T15:44:08.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 432997 4.00 MB/sec execute 464 sec latency 9.778 ms 2024-09-13T15:44:08.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1282830 11.92 MB/sec execute 464 sec latency 33.544 ms 2024-09-13T15:44:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:07 smithi090 ceph-mon[27811]: pgmap v564: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-13T15:44:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:07 smithi175 ceph-mon[36128]: pgmap v564: 129 pgs: 129 active+clean; 179 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-13T15:44:09.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 433593 3.99 MB/sec execute 465 sec latency 10.745 ms 2024-09-13T15:44:09.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1286304 11.93 MB/sec execute 465 sec latency 94.273 ms 2024-09-13T15:44:10.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 434415 3.99 MB/sec execute 466 sec latency 16.059 ms 2024-09-13T15:44:10.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1289796 11.95 MB/sec execute 466 sec latency 33.415 ms 2024-09-13T15:44:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:09 smithi090 ceph-mon[27811]: pgmap v565: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.2 MiB/s wr, 107 op/s 2024-09-13T15:44:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:09 smithi175 ceph-mon[36128]: pgmap v565: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.2 MiB/s wr, 107 op/s 2024-09-13T15:44:11.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 435220 3.99 MB/sec execute 467 sec latency 20.822 ms 2024-09-13T15:44:11.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1292314 11.95 MB/sec execute 467 sec latency 33.481 ms 2024-09-13T15:44:12.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 436105 3.99 MB/sec execute 468 sec latency 10.985 ms 2024-09-13T15:44:12.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1294302 11.95 MB/sec execute 468 sec latency 132.992 ms 2024-09-13T15:44:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:11 smithi090 ceph-mon[27811]: pgmap v566: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-13T15:44:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:11 smithi175 ceph-mon[36128]: pgmap v566: 129 pgs: 129 active+clean; 181 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 88 op/s 2024-09-13T15:44:13.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 436791 3.99 MB/sec execute 469 sec latency 10.988 ms 2024-09-13T15:44:13.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1297177 11.96 MB/sec execute 469 sec latency 32.900 ms 2024-09-13T15:44:14.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 437568 3.99 MB/sec execute 470 sec latency 13.102 ms 2024-09-13T15:44:14.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1300537 11.97 MB/sec execute 470 sec latency 121.192 ms 2024-09-13T15:44:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:13 smithi090 ceph-mon[27811]: pgmap v567: 129 pgs: 129 active+clean; 182 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-09-13T15:44:14.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:13 smithi175 ceph-mon[36128]: pgmap v567: 129 pgs: 129 active+clean; 182 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.0 MiB/s wr, 96 op/s 2024-09-13T15:44:15.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 438372 3.99 MB/sec execute 471 sec latency 113.725 ms 2024-09-13T15:44:15.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1302295 11.96 MB/sec execute 471 sec latency 356.093 ms 2024-09-13T15:44:16.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 439323 4.00 MB/sec execute 472 sec latency 10.676 ms 2024-09-13T15:44:16.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1304357 11.96 MB/sec execute 472 sec latency 166.675 ms 2024-09-13T15:44:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:15 smithi090 ceph-mon[27811]: pgmap v568: 129 pgs: 129 active+clean; 183 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.3 MiB/s wr, 117 op/s 2024-09-13T15:44:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:15 smithi175 ceph-mon[36128]: pgmap v568: 129 pgs: 129 active+clean; 183 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 2.3 MiB/s wr, 117 op/s 2024-09-13T15:44:17.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 440112 4.00 MB/sec execute 473 sec latency 10.545 ms 2024-09-13T15:44:17.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1307710 11.98 MB/sec execute 473 sec latency 137.663 ms 2024-09-13T15:44:18.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 440719 3.99 MB/sec execute 474 sec latency 10.833 ms 2024-09-13T15:44:18.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1311192 11.99 MB/sec execute 474 sec latency 33.587 ms 2024-09-13T15:44:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:17 smithi090 ceph-mon[27811]: pgmap v569: 129 pgs: 129 active+clean; 183 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-09-13T15:44:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:17 smithi175 ceph-mon[36128]: pgmap v569: 129 pgs: 129 active+clean; 183 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.0 MiB/s wr, 92 op/s 2024-09-13T15:44:19.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 441523 4.00 MB/sec execute 475 sec latency 12.456 ms 2024-09-13T15:44:19.217 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1314681 12.00 MB/sec execute 475 sec latency 33.450 ms 2024-09-13T15:44:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:20.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:19 smithi090 ceph-mon[27811]: pgmap v570: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-09-13T15:44:20.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 442327 3.99 MB/sec execute 476 sec latency 11.534 ms 2024-09-13T15:44:20.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1316764 12.00 MB/sec execute 476 sec latency 145.015 ms 2024-09-13T15:44:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:19 smithi175 ceph-mon[36128]: pgmap v570: 129 pgs: 129 active+clean; 183 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.1 MiB/s wr, 97 op/s 2024-09-13T15:44:21.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 443146 4.00 MB/sec execute 477 sec latency 179.342 ms 2024-09-13T15:44:21.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1316977 11.98 MB/sec execute 477 sec latency 432.107 ms 2024-09-13T15:44:22.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 443645 4.00 MB/sec execute 478 sec latency 132.234 ms 2024-09-13T15:44:22.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1317292 11.96 MB/sec execute 478 sec latency 350.949 ms 2024-09-13T15:44:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:21 smithi090 ceph-mon[27811]: pgmap v571: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:44:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:21 smithi175 ceph-mon[36128]: pgmap v571: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:44:23.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 444062 3.99 MB/sec execute 479 sec latency 128.382 ms 2024-09-13T15:44:23.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1317334 11.93 MB/sec execute 479 sec latency 357.473 ms 2024-09-13T15:44:24.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 444792 3.99 MB/sec execute 480 sec latency 52.913 ms 2024-09-13T15:44:24.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318228 11.92 MB/sec execute 480 sec latency 370.145 ms 2024-09-13T15:44:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:23 smithi090 ceph-mon[27811]: pgmap v572: 129 pgs: 129 active+clean; 184 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-13T15:44:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:23 smithi175 ceph-mon[36128]: pgmap v572: 129 pgs: 129 active+clean; 184 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 92 op/s 2024-09-13T15:44:25.020 INFO:tasks.workunit.client.1.smithi175.stdout: 1 445393 3.99 MB/sec execute 481 sec latency 136.561 ms 2024-09-13T15:44:25.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318235 11.89 MB/sec execute 481 sec latency 380.261 ms 2024-09-13T15:44:26.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 445900 3.98 MB/sec execute 482 sec latency 138.936 ms 2024-09-13T15:44:26.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318281 11.87 MB/sec execute 482 sec latency 370.958 ms 2024-09-13T15:44:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:25 smithi090 ceph-mon[27811]: pgmap v573: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-13T15:44:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:25 smithi175 ceph-mon[36128]: pgmap v573: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 95 op/s 2024-09-13T15:44:27.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 446713 3.99 MB/sec execute 483 sec latency 145.318 ms 2024-09-13T15:44:27.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318288 11.85 MB/sec execute 483 sec latency 367.898 ms 2024-09-13T15:44:27.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:44:27.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:44:27.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:44:27.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:44:27.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:44:27.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:44:27.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:44:27.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:44:28.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 447202 3.99 MB/sec execute 484 sec latency 153.358 ms 2024-09-13T15:44:28.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318349 11.82 MB/sec execute 484 sec latency 459.053 ms 2024-09-13T15:44:28.236 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:27 smithi175 ceph-mon[36128]: pgmap v574: 129 pgs: 129 active+clean; 186 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-13T15:44:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:27 smithi090 ceph-mon[27811]: pgmap v574: 129 pgs: 129 active+clean; 186 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-13T15:44:29.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 447560 3.98 MB/sec execute 485 sec latency 147.956 ms 2024-09-13T15:44:29.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318354 11.80 MB/sec execute 485 sec latency 422.639 ms 2024-09-13T15:44:30.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 448018 3.98 MB/sec execute 486 sec latency 144.411 ms 2024-09-13T15:44:30.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1318430 11.77 MB/sec execute 486 sec latency 400.863 ms 2024-09-13T15:44:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:29 smithi090 ceph-mon[27811]: pgmap v575: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:44:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:29 smithi175 ceph-mon[36128]: pgmap v575: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:44:31.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 448724 3.98 MB/sec execute 487 sec latency 77.266 ms 2024-09-13T15:44:31.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1319439 11.76 MB/sec execute 487 sec latency 399.942 ms 2024-09-13T15:44:32.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 449526 3.97 MB/sec execute 488 sec latency 11.380 ms 2024-09-13T15:44:32.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1322081 11.76 MB/sec execute 488 sec latency 56.597 ms 2024-09-13T15:44:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:31 smithi090 ceph-mon[27811]: pgmap v576: 129 pgs: 129 active+clean; 186 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-09-13T15:44:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:31 smithi175 ceph-mon[36128]: pgmap v576: 129 pgs: 129 active+clean; 186 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.4 MiB/s wr, 80 op/s 2024-09-13T15:44:33.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 450335 3.98 MB/sec execute 489 sec latency 11.334 ms 2024-09-13T15:44:33.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1325560 11.78 MB/sec execute 489 sec latency 33.254 ms 2024-09-13T15:44:34.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 451006 3.98 MB/sec execute 490 sec latency 10.052 ms 2024-09-13T15:44:34.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1328989 11.79 MB/sec execute 490 sec latency 33.517 ms 2024-09-13T15:44:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:33 smithi090 ceph-mon[27811]: pgmap v577: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-09-13T15:44:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:33 smithi175 ceph-mon[36128]: pgmap v577: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.5 MiB/s wr, 82 op/s 2024-09-13T15:44:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:35.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 451733 3.98 MB/sec execute 491 sec latency 51.116 ms 2024-09-13T15:44:35.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1332529 11.81 MB/sec execute 491 sec latency 33.442 ms 2024-09-13T15:44:36.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 452646 3.97 MB/sec execute 492 sec latency 20.140 ms 2024-09-13T15:44:36.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1336013 11.82 MB/sec execute 492 sec latency 33.258 ms 2024-09-13T15:44:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:35 smithi090 ceph-mon[27811]: pgmap v578: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-13T15:44:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:35 smithi175 ceph-mon[36128]: pgmap v578: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-13T15:44:37.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 453430 3.98 MB/sec execute 493 sec latency 84.573 ms 2024-09-13T15:44:37.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1336158 11.80 MB/sec execute 493 sec latency 489.345 ms 2024-09-13T15:44:38.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 454059 3.98 MB/sec execute 494 sec latency 99.306 ms 2024-09-13T15:44:38.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1336198 11.78 MB/sec execute 494 sec latency 391.759 ms 2024-09-13T15:44:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:37 smithi090 ceph-mon[27811]: pgmap v579: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 92 op/s 2024-09-13T15:44:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:37 smithi175 ceph-mon[36128]: pgmap v579: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 92 op/s 2024-09-13T15:44:39.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 454626 3.98 MB/sec execute 495 sec latency 61.416 ms 2024-09-13T15:44:39.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1338612 11.78 MB/sec execute 495 sec latency 375.592 ms 2024-09-13T15:44:40.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 455445 3.98 MB/sec execute 496 sec latency 14.323 ms 2024-09-13T15:44:40.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1341799 11.79 MB/sec execute 496 sec latency 72.196 ms 2024-09-13T15:44:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:39 smithi090 ceph-mon[27811]: pgmap v580: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:44:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:39 smithi175 ceph-mon[36128]: pgmap v580: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.7 MiB/s wr, 84 op/s 2024-09-13T15:44:41.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 456330 3.98 MB/sec execute 497 sec latency 11.219 ms 2024-09-13T15:44:41.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1344780 11.80 MB/sec execute 497 sec latency 33.419 ms 2024-09-13T15:44:42.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 457325 3.98 MB/sec execute 498 sec latency 10.581 ms 2024-09-13T15:44:42.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1347802 11.81 MB/sec execute 498 sec latency 83.558 ms 2024-09-13T15:44:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:41 smithi090 ceph-mon[27811]: pgmap v581: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-13T15:44:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:41 smithi175 ceph-mon[36128]: pgmap v581: 129 pgs: 129 active+clean; 188 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.5 MiB/s wr, 76 op/s 2024-09-13T15:44:43.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 457925 3.98 MB/sec execute 499 sec latency 11.409 ms 2024-09-13T15:44:43.218 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1350340 11.81 MB/sec execute 499 sec latency 125.386 ms 2024-09-13T15:44:44.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 458465 3.99 MB/sec execute 500 sec latency 52.478 ms 2024-09-13T15:44:44.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1353740 11.83 MB/sec execute 500 sec latency 136.523 ms 2024-09-13T15:44:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:43 smithi090 ceph-mon[27811]: pgmap v582: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-13T15:44:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:43 smithi175 ceph-mon[36128]: pgmap v582: 129 pgs: 129 active+clean; 189 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 87 op/s 2024-09-13T15:44:45.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 459390 4.01 MB/sec execute 501 sec latency 92.197 ms 2024-09-13T15:44:45.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1353801 11.80 MB/sec execute 501 sec latency 406.924 ms 2024-09-13T15:44:46.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 460145 4.01 MB/sec execute 502 sec latency 117.233 ms 2024-09-13T15:44:46.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1353806 11.78 MB/sec execute 502 sec latency 403.821 ms 2024-09-13T15:44:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:45 smithi090 ceph-mon[27811]: pgmap v583: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 92 op/s 2024-09-13T15:44:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:45 smithi175 ceph-mon[36128]: pgmap v583: 129 pgs: 129 active+clean; 190 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 92 op/s 2024-09-13T15:44:47.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 460814 4.01 MB/sec execute 503 sec latency 109.500 ms 2024-09-13T15:44:47.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1355921 11.78 MB/sec execute 503 sec latency 371.233 ms 2024-09-13T15:44:48.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 461679 4.02 MB/sec execute 504 sec latency 11.990 ms 2024-09-13T15:44:48.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1358921 11.78 MB/sec execute 504 sec latency 96.478 ms 2024-09-13T15:44:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:47 smithi090 ceph-mon[27811]: pgmap v584: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.2 MiB/s wr, 91 op/s 2024-09-13T15:44:48.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:47 smithi175 ceph-mon[36128]: pgmap v584: 129 pgs: 129 active+clean; 185 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.2 MiB/s wr, 91 op/s 2024-09-13T15:44:49.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 462375 4.02 MB/sec execute 505 sec latency 9.622 ms 2024-09-13T15:44:49.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1361085 11.79 MB/sec execute 505 sec latency 152.025 ms 2024-09-13T15:44:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:44:50.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 463089 4.01 MB/sec execute 506 sec latency 10.933 ms 2024-09-13T15:44:50.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1364624 11.80 MB/sec execute 506 sec latency 40.315 ms 2024-09-13T15:44:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:49 smithi090 ceph-mon[27811]: pgmap v585: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.9 MiB/s wr, 97 op/s 2024-09-13T15:44:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:49 smithi175 ceph-mon[36128]: pgmap v585: 129 pgs: 129 active+clean; 187 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.9 MiB/s wr, 97 op/s 2024-09-13T15:44:51.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 463987 4.01 MB/sec execute 507 sec latency 15.580 ms 2024-09-13T15:44:51.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1367988 11.81 MB/sec execute 507 sec latency 129.663 ms 2024-09-13T15:44:52.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 464830 4.02 MB/sec execute 508 sec latency 99.994 ms 2024-09-13T15:44:52.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1370102 11.81 MB/sec execute 508 sec latency 357.905 ms 2024-09-13T15:44:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:51 smithi090 ceph-mon[27811]: pgmap v586: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 MiB/s wr, 90 op/s 2024-09-13T15:44:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:51 smithi175 ceph-mon[36128]: pgmap v586: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.2 MiB/s wr, 90 op/s 2024-09-13T15:44:53.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 465534 4.02 MB/sec execute 509 sec latency 124.607 ms 2024-09-13T15:44:53.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1370377 11.79 MB/sec execute 509 sec latency 381.992 ms 2024-09-13T15:44:54.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 465935 4.01 MB/sec execute 510 sec latency 138.911 ms 2024-09-13T15:44:54.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1371470 11.78 MB/sec execute 510 sec latency 378.572 ms 2024-09-13T15:44:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:53 smithi090 ceph-mon[27811]: pgmap v587: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.5 MiB/s wr, 99 op/s 2024-09-13T15:44:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:53 smithi175 ceph-mon[36128]: pgmap v587: 129 pgs: 129 active+clean; 191 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.5 MiB/s wr, 99 op/s 2024-09-13T15:44:55.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 466707 4.01 MB/sec execute 511 sec latency 10.451 ms 2024-09-13T15:44:55.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1375698 11.84 MB/sec execute 511 sec latency 33.314 ms 2024-09-13T15:44:56.021 INFO:tasks.workunit.client.1.smithi175.stdout: 1 467623 4.01 MB/sec execute 512 sec latency 13.634 ms 2024-09-13T15:44:56.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1378371 11.84 MB/sec execute 512 sec latency 32.409 ms 2024-09-13T15:44:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:55 smithi090 ceph-mon[27811]: pgmap v588: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.6 MiB/s wr, 113 op/s 2024-09-13T15:44:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:55 smithi175 ceph-mon[36128]: pgmap v588: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.6 MiB/s wr, 113 op/s 2024-09-13T15:44:57.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 468709 4.02 MB/sec execute 513 sec latency 9.452 ms 2024-09-13T15:44:57.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1381587 11.85 MB/sec execute 513 sec latency 32.558 ms 2024-09-13T15:44:58.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 469294 4.02 MB/sec execute 514 sec latency 10.651 ms 2024-09-13T15:44:58.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1384481 11.85 MB/sec execute 514 sec latency 103.300 ms 2024-09-13T15:44:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:57 smithi090 ceph-mon[27811]: pgmap v589: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 93 op/s 2024-09-13T15:44:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:57 smithi175 ceph-mon[36128]: pgmap v589: 129 pgs: 129 active+clean; 192 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.3 MiB/s wr, 93 op/s 2024-09-13T15:44:59.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 469926 4.01 MB/sec execute 515 sec latency 11.786 ms 2024-09-13T15:44:59.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1386413 11.86 MB/sec execute 515 sec latency 318.346 ms 2024-09-13T15:45:00.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 470640 4.02 MB/sec execute 516 sec latency 68.085 ms 2024-09-13T15:45:00.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1386420 11.83 MB/sec execute 516 sec latency 360.216 ms 2024-09-13T15:45:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:44:59 smithi090 ceph-mon[27811]: pgmap v590: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.2 MiB/s wr, 112 op/s 2024-09-13T15:45:00.269 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:44:59 smithi175 ceph-mon[36128]: pgmap v590: 129 pgs: 129 active+clean; 194 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 3.2 MiB/s wr, 112 op/s 2024-09-13T15:45:01.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 471362 4.01 MB/sec execute 517 sec latency 122.087 ms 2024-09-13T15:45:01.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1386670 11.81 MB/sec execute 517 sec latency 360.388 ms 2024-09-13T15:45:02.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 472305 4.02 MB/sec execute 518 sec latency 11.547 ms 2024-09-13T15:45:02.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1390040 11.82 MB/sec execute 518 sec latency 33.401 ms 2024-09-13T15:45:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:01 smithi090 ceph-mon[27811]: pgmap v591: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.3 MiB/s wr, 82 op/s 2024-09-13T15:45:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:01 smithi175 ceph-mon[36128]: pgmap v591: 129 pgs: 129 active+clean; 194 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.3 MiB/s wr, 82 op/s 2024-09-13T15:45:03.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 472949 4.02 MB/sec execute 519 sec latency 13.292 ms 2024-09-13T15:45:03.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1393583 11.84 MB/sec execute 519 sec latency 56.832 ms 2024-09-13T15:45:04.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 473648 4.01 MB/sec execute 520 sec latency 10.560 ms 2024-09-13T15:45:04.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1397159 11.85 MB/sec execute 520 sec latency 40.157 ms 2024-09-13T15:45:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:04 smithi175 ceph-mon[36128]: pgmap v592: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:45:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:04.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:04 smithi090 ceph-mon[27811]: pgmap v592: 129 pgs: 129 active+clean; 193 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 86 op/s 2024-09-13T15:45:04.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:05.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 474518 4.01 MB/sec execute 521 sec latency 20.878 ms 2024-09-13T15:45:05.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1400641 11.87 MB/sec execute 521 sec latency 65.436 ms 2024-09-13T15:45:06.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 475437 4.02 MB/sec execute 522 sec latency 10.570 ms 2024-09-13T15:45:06.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1404181 11.88 MB/sec execute 522 sec latency 31.726 ms 2024-09-13T15:45:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:06 smithi090 ceph-mon[27811]: pgmap v593: 129 pgs: 129 active+clean; 195 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 104 op/s 2024-09-13T15:45:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:06 smithi175 ceph-mon[36128]: pgmap v593: 129 pgs: 129 active+clean; 195 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 104 op/s 2024-09-13T15:45:07.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 476229 4.02 MB/sec execute 523 sec latency 12.397 ms 2024-09-13T15:45:07.241 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1407680 11.89 MB/sec execute 523 sec latency 33.418 ms 2024-09-13T15:45:07.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:07 smithi090 ceph-mon[27811]: pgmap v594: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:45:07.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:07 smithi175 ceph-mon[36128]: pgmap v594: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:45:08.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 476794 4.02 MB/sec execute 524 sec latency 11.072 ms 2024-09-13T15:45:08.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1411221 11.91 MB/sec execute 524 sec latency 33.468 ms 2024-09-13T15:45:09.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 477623 4.02 MB/sec execute 525 sec latency 13.337 ms 2024-09-13T15:45:09.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1414205 11.92 MB/sec execute 525 sec latency 41.876 ms 2024-09-13T15:45:10.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 478520 4.02 MB/sec execute 526 sec latency 19.991 ms 2024-09-13T15:45:10.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1417390 11.93 MB/sec execute 526 sec latency 32.743 ms 2024-09-13T15:45:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:09 smithi090 ceph-mon[27811]: pgmap v595: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-09-13T15:45:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:09 smithi175 ceph-mon[36128]: pgmap v595: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 108 op/s 2024-09-13T15:45:11.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 479438 4.02 MB/sec execute 527 sec latency 12.792 ms 2024-09-13T15:45:11.219 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1420609 11.94 MB/sec execute 527 sec latency 50.157 ms 2024-09-13T15:45:11.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:11 smithi090 ceph-mon[27811]: pgmap v596: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:45:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:11 smithi175 ceph-mon[36128]: pgmap v596: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:45:12.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 480090 4.02 MB/sec execute 528 sec latency 12.702 ms 2024-09-13T15:45:12.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1423657 11.94 MB/sec execute 528 sec latency 33.519 ms 2024-09-13T15:45:13.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 480759 4.02 MB/sec execute 529 sec latency 55.932 ms 2024-09-13T15:45:13.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1426210 11.95 MB/sec execute 529 sec latency 166.971 ms 2024-09-13T15:45:14.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 481594 4.02 MB/sec execute 530 sec latency 12.447 ms 2024-09-13T15:45:14.052 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:13 smithi175 ceph-mon[36128]: pgmap v597: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-13T15:45:14.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1429152 11.95 MB/sec execute 530 sec latency 108.555 ms 2024-09-13T15:45:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:13 smithi090 ceph-mon[27811]: pgmap v597: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.8 MiB/s wr, 93 op/s 2024-09-13T15:45:15.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 482465 4.02 MB/sec execute 531 sec latency 24.833 ms 2024-09-13T15:45:15.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1432545 11.97 MB/sec execute 531 sec latency 175.116 ms 2024-09-13T15:45:16.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 483332 4.03 MB/sec execute 532 sec latency 9.726 ms 2024-09-13T15:45:16.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1436042 11.98 MB/sec execute 532 sec latency 33.131 ms 2024-09-13T15:45:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:15 smithi090 ceph-mon[27811]: pgmap v598: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:45:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:15 smithi175 ceph-mon[36128]: pgmap v598: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.2 MiB/s wr, 115 op/s 2024-09-13T15:45:17.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 483896 4.02 MB/sec execute 533 sec latency 10.179 ms 2024-09-13T15:45:17.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1438654 11.98 MB/sec execute 533 sec latency 49.658 ms 2024-09-13T15:45:18.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 484742 4.02 MB/sec execute 534 sec latency 10.610 ms 2024-09-13T15:45:18.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1441669 11.99 MB/sec execute 534 sec latency 41.719 ms 2024-09-13T15:45:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:17 smithi090 ceph-mon[27811]: pgmap v599: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 94 op/s 2024-09-13T15:45:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:17 smithi175 ceph-mon[36128]: pgmap v599: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 94 op/s 2024-09-13T15:45:19.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 485603 4.02 MB/sec execute 535 sec latency 20.513 ms 2024-09-13T15:45:19.143 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:19.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1444798 12.00 MB/sec execute 535 sec latency 32.224 ms 2024-09-13T15:45:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:20.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 486526 4.03 MB/sec execute 536 sec latency 13.757 ms 2024-09-13T15:45:20.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1447633 12.01 MB/sec execute 536 sec latency 33.430 ms 2024-09-13T15:45:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:19 smithi090 ceph-mon[27811]: pgmap v600: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 117 op/s 2024-09-13T15:45:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:19 smithi175 ceph-mon[36128]: pgmap v600: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 2.0 MiB/s wr, 117 op/s 2024-09-13T15:45:21.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 487171 4.03 MB/sec execute 537 sec latency 10.123 ms 2024-09-13T15:45:21.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1450381 12.01 MB/sec execute 537 sec latency 201.338 ms 2024-09-13T15:45:22.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 487851 4.02 MB/sec execute 538 sec latency 43.797 ms 2024-09-13T15:45:22.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1452556 12.01 MB/sec execute 538 sec latency 298.951 ms 2024-09-13T15:45:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:21 smithi090 ceph-mon[27811]: pgmap v601: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-13T15:45:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:21 smithi175 ceph-mon[36128]: pgmap v601: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.9 MiB/s wr, 88 op/s 2024-09-13T15:45:23.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 488741 4.02 MB/sec execute 539 sec latency 16.152 ms 2024-09-13T15:45:23.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1455597 12.02 MB/sec execute 539 sec latency 33.511 ms 2024-09-13T15:45:24.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 489670 4.03 MB/sec execute 540 sec latency 14.385 ms 2024-09-13T15:45:24.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1458561 12.03 MB/sec execute 540 sec latency 25.256 ms 2024-09-13T15:45:24.237 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:23 smithi175 ceph-mon[36128]: pgmap v602: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 99 op/s 2024-09-13T15:45:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:23 smithi090 ceph-mon[27811]: pgmap v602: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.8 MiB/s wr, 99 op/s 2024-09-13T15:45:25.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 490445 4.03 MB/sec execute 541 sec latency 10.887 ms 2024-09-13T15:45:25.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1461846 12.04 MB/sec execute 541 sec latency 33.564 ms 2024-09-13T15:45:26.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 491028 4.03 MB/sec execute 542 sec latency 9.390 ms 2024-09-13T15:45:26.215 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:25 smithi090 ceph-mon[27811]: pgmap v603: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.2 MiB/s wr, 109 op/s 2024-09-13T15:45:26.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1464565 12.04 MB/sec execute 542 sec latency 178.777 ms 2024-09-13T15:45:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:25 smithi175 ceph-mon[36128]: pgmap v603: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.2 MiB/s wr, 109 op/s 2024-09-13T15:45:27.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 491817 4.03 MB/sec execute 543 sec latency 11.969 ms 2024-09-13T15:45:27.158 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:26 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:45:27.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1467083 12.05 MB/sec execute 543 sec latency 349.086 ms 2024-09-13T15:45:27.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:26 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:45:28.022 INFO:tasks.workunit.client.1.smithi175.stdout: 1 492617 4.02 MB/sec execute 544 sec latency 57.865 ms 2024-09-13T15:45:28.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1467980 12.03 MB/sec execute 544 sec latency 384.250 ms 2024-09-13T15:45:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:27 smithi090 ceph-mon[27811]: pgmap v604: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 103 op/s 2024-09-13T15:45:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:27 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:45:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:27 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:45:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:27 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:45:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:27 smithi175 ceph-mon[36128]: pgmap v604: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 103 op/s 2024-09-13T15:45:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:27 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:45:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:27 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:45:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:27 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:45:29.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 493330 4.03 MB/sec execute 545 sec latency 120.939 ms 2024-09-13T15:45:29.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1467987 12.01 MB/sec execute 545 sec latency 378.615 ms 2024-09-13T15:45:30.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 493931 4.03 MB/sec execute 546 sec latency 110.495 ms 2024-09-13T15:45:30.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468034 11.99 MB/sec execute 546 sec latency 450.901 ms 2024-09-13T15:45:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:29 smithi090 ceph-mon[27811]: pgmap v605: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.9 MiB/s wr, 107 op/s 2024-09-13T15:45:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:29 smithi175 ceph-mon[36128]: pgmap v605: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 1.9 MiB/s wr, 107 op/s 2024-09-13T15:45:31.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 494287 4.02 MB/sec execute 547 sec latency 119.956 ms 2024-09-13T15:45:31.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468041 11.97 MB/sec execute 547 sec latency 433.850 ms 2024-09-13T15:45:32.025 INFO:tasks.workunit.client.1.smithi175.stdout: 1 494921 4.02 MB/sec execute 548 sec latency 63.396 ms 2024-09-13T15:45:32.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468046 11.95 MB/sec execute 548 sec latency 362.675 ms 2024-09-13T15:45:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:31 smithi090 ceph-mon[27811]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-13T15:45:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:31 smithi175 ceph-mon[36128]: pgmap v606: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.7 MiB/s wr, 79 op/s 2024-09-13T15:45:33.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 495657 4.02 MB/sec execute 549 sec latency 130.182 ms 2024-09-13T15:45:33.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468107 11.92 MB/sec execute 549 sec latency 360.425 ms 2024-09-13T15:45:34.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 496226 4.01 MB/sec execute 550 sec latency 136.836 ms 2024-09-13T15:45:34.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468114 11.90 MB/sec execute 550 sec latency 371.521 ms 2024-09-13T15:45:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:33 smithi090 ceph-mon[27811]: pgmap v607: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.8 MiB/s wr, 99 op/s 2024-09-13T15:45:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:33 smithi175 ceph-mon[36128]: pgmap v607: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.8 KiB/s rd, 1.8 MiB/s wr, 99 op/s 2024-09-13T15:45:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:35.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 497024 4.02 MB/sec execute 551 sec latency 107.179 ms 2024-09-13T15:45:35.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468225 11.88 MB/sec execute 551 sec latency 359.550 ms 2024-09-13T15:45:36.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 497417 4.02 MB/sec execute 552 sec latency 142.731 ms 2024-09-13T15:45:36.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468294 11.86 MB/sec execute 552 sec latency 368.188 ms 2024-09-13T15:45:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:35 smithi090 ceph-mon[27811]: pgmap v608: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-13T15:45:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:35 smithi175 ceph-mon[36128]: pgmap v608: 129 pgs: 129 active+clean; 196 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.4 KiB/s rd, 1.9 MiB/s wr, 91 op/s 2024-09-13T15:45:37.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 497854 4.01 MB/sec execute 553 sec latency 145.093 ms 2024-09-13T15:45:37.224 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1468941 11.85 MB/sec execute 553 sec latency 374.876 ms 2024-09-13T15:45:38.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 498564 4.01 MB/sec execute 554 sec latency 78.940 ms 2024-09-13T15:45:38.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1471593 11.85 MB/sec execute 554 sec latency 292.013 ms 2024-09-13T15:45:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:37 smithi090 ceph-mon[27811]: pgmap v609: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:45:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:37 smithi175 ceph-mon[36128]: pgmap v609: 129 pgs: 129 active+clean; 197 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.7 MiB/s wr, 89 op/s 2024-09-13T15:45:39.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 499439 4.01 MB/sec execute 555 sec latency 11.894 ms 2024-09-13T15:45:39.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1475082 11.86 MB/sec execute 555 sec latency 60.011 ms 2024-09-13T15:45:40.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 500332 4.01 MB/sec execute 556 sec latency 17.016 ms 2024-09-13T15:45:40.220 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1478618 11.88 MB/sec execute 556 sec latency 33.541 ms 2024-09-13T15:45:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:39 smithi090 ceph-mon[27811]: pgmap v610: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:45:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:39 smithi175 ceph-mon[36128]: pgmap v610: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.7 MiB/s wr, 82 op/s 2024-09-13T15:45:41.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 501116 4.02 MB/sec execute 557 sec latency 13.997 ms 2024-09-13T15:45:41.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1480939 11.88 MB/sec execute 557 sec latency 41.208 ms 2024-09-13T15:45:42.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:41 smithi090 ceph-mon[27811]: pgmap v611: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-09-13T15:45:42.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 501681 4.01 MB/sec execute 558 sec latency 13.188 ms 2024-09-13T15:45:42.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1483961 11.88 MB/sec execute 558 sec latency 33.444 ms 2024-09-13T15:45:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:41 smithi175 ceph-mon[36128]: pgmap v611: 129 pgs: 129 active+clean; 198 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.6 MiB/s wr, 73 op/s 2024-09-13T15:45:43.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 502365 4.01 MB/sec execute 559 sec latency 93.439 ms 2024-09-13T15:45:43.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1485709 11.88 MB/sec execute 559 sec latency 351.380 ms 2024-09-13T15:45:44.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 503084 4.01 MB/sec execute 560 sec latency 88.984 ms 2024-09-13T15:45:44.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1485716 11.86 MB/sec execute 560 sec latency 368.231 ms 2024-09-13T15:45:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:43 smithi090 ceph-mon[27811]: pgmap v612: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:45:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:43 smithi175 ceph-mon[36128]: pgmap v612: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:45:45.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 503870 4.01 MB/sec execute 561 sec latency 112.993 ms 2024-09-13T15:45:45.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1486963 11.85 MB/sec execute 561 sec latency 357.079 ms 2024-09-13T15:45:46.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 504699 4.02 MB/sec execute 562 sec latency 8.921 ms 2024-09-13T15:45:46.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1489499 11.85 MB/sec execute 562 sec latency 123.326 ms 2024-09-13T15:45:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:45 smithi090 ceph-mon[27811]: pgmap v613: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:45:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:45 smithi175 ceph-mon[36128]: pgmap v613: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.8 MiB/s wr, 85 op/s 2024-09-13T15:45:47.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 505249 4.01 MB/sec execute 563 sec latency 11.611 ms 2024-09-13T15:45:47.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1492931 11.86 MB/sec execute 563 sec latency 33.109 ms 2024-09-13T15:45:48.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 506086 4.01 MB/sec execute 564 sec latency 12.938 ms 2024-09-13T15:45:48.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1496416 11.88 MB/sec execute 564 sec latency 33.452 ms 2024-09-13T15:45:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:47 smithi090 ceph-mon[27811]: pgmap v614: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:45:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:47 smithi175 ceph-mon[36128]: pgmap v614: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:45:49.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 506923 4.01 MB/sec execute 565 sec latency 14.948 ms 2024-09-13T15:45:49.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:49.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1499949 11.89 MB/sec execute 565 sec latency 33.303 ms 2024-09-13T15:45:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:45:50.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:49 smithi175 ceph-mon[36128]: pgmap v615: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:45:50.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 507809 4.01 MB/sec execute 566 sec latency 11.568 ms 2024-09-13T15:45:50.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1503442 11.90 MB/sec execute 566 sec latency 43.244 ms 2024-09-13T15:45:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:49 smithi090 ceph-mon[27811]: pgmap v615: 129 pgs: 129 active+clean; 200 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.0 KiB/s rd, 1.7 MiB/s wr, 92 op/s 2024-09-13T15:45:51.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 508466 4.02 MB/sec execute 567 sec latency 12.187 ms 2024-09-13T15:45:51.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1506097 11.91 MB/sec execute 567 sec latency 33.398 ms 2024-09-13T15:45:52.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 509153 4.01 MB/sec execute 568 sec latency 13.577 ms 2024-09-13T15:45:52.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1508344 11.90 MB/sec execute 568 sec latency 191.746 ms 2024-09-13T15:45:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:51 smithi090 ceph-mon[27811]: pgmap v616: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:45:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:51 smithi175 ceph-mon[36128]: pgmap v616: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:45:53.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 510014 4.01 MB/sec execute 569 sec latency 13.843 ms 2024-09-13T15:45:53.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1511490 11.92 MB/sec execute 569 sec latency 33.420 ms 2024-09-13T15:45:54.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 510911 4.01 MB/sec execute 570 sec latency 21.410 ms 2024-09-13T15:45:54.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1514326 11.92 MB/sec execute 570 sec latency 33.218 ms 2024-09-13T15:45:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:53 smithi090 ceph-mon[27811]: pgmap v617: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-13T15:45:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:53 smithi175 ceph-mon[36128]: pgmap v617: 129 pgs: 129 active+clean; 201 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 1.8 MiB/s wr, 87 op/s 2024-09-13T15:45:55.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 511403 4.01 MB/sec execute 571 sec latency 195.140 ms 2024-09-13T15:45:55.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1516380 11.92 MB/sec execute 571 sec latency 480.082 ms 2024-09-13T15:45:56.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 511958 4.01 MB/sec execute 572 sec latency 132.505 ms 2024-09-13T15:45:56.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1516595 11.90 MB/sec execute 572 sec latency 369.383 ms 2024-09-13T15:45:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:55 smithi090 ceph-mon[27811]: pgmap v618: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-13T15:45:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:55 smithi175 ceph-mon[36128]: pgmap v618: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.0 MiB/s wr, 104 op/s 2024-09-13T15:45:57.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 512404 4.01 MB/sec execute 573 sec latency 147.535 ms 2024-09-13T15:45:57.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1516726 11.88 MB/sec execute 573 sec latency 387.044 ms 2024-09-13T15:45:58.023 INFO:tasks.workunit.client.1.smithi175.stdout: 1 513035 4.01 MB/sec execute 574 sec latency 151.824 ms 2024-09-13T15:45:58.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1517621 11.87 MB/sec execute 574 sec latency 365.392 ms 2024-09-13T15:45:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:57 smithi090 ceph-mon[27811]: pgmap v619: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-13T15:45:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:57 smithi175 ceph-mon[36128]: pgmap v619: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 83 op/s 2024-09-13T15:45:59.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 513676 4.00 MB/sec execute 575 sec latency 123.386 ms 2024-09-13T15:45:59.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1517677 11.85 MB/sec execute 575 sec latency 297.721 ms 2024-09-13T15:46:00.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 514199 4.00 MB/sec execute 576 sec latency 174.292 ms 2024-09-13T15:46:00.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1517936 11.83 MB/sec execute 576 sec latency 464.853 ms 2024-09-13T15:46:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:45:59 smithi090 ceph-mon[27811]: pgmap v620: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-13T15:46:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:45:59 smithi175 ceph-mon[36128]: pgmap v620: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.9 KiB/s rd, 1.8 MiB/s wr, 102 op/s 2024-09-13T15:46:01.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 514956 4.00 MB/sec execute 577 sec latency 264.913 ms 2024-09-13T15:46:01.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1521378 11.84 MB/sec execute 577 sec latency 33.478 ms 2024-09-13T15:46:02.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 515609 4.01 MB/sec execute 578 sec latency 10.627 ms 2024-09-13T15:46:02.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1524835 11.85 MB/sec execute 578 sec latency 67.100 ms 2024-09-13T15:46:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:01 smithi090 ceph-mon[27811]: pgmap v621: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:46:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:01 smithi175 ceph-mon[36128]: pgmap v621: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.7 MiB/s wr, 75 op/s 2024-09-13T15:46:03.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 516346 4.00 MB/sec execute 579 sec latency 13.389 ms 2024-09-13T15:46:03.221 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1528322 11.87 MB/sec execute 579 sec latency 33.202 ms 2024-09-13T15:46:04.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 517223 4.00 MB/sec execute 580 sec latency 11.935 ms 2024-09-13T15:46:04.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1531813 11.88 MB/sec execute 580 sec latency 31.913 ms 2024-09-13T15:46:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:03 smithi090 ceph-mon[27811]: pgmap v622: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:46:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:03 smithi175 ceph-mon[36128]: pgmap v622: 129 pgs: 129 active+clean; 199 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.4 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:46:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:05.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 517703 4.00 MB/sec execute 581 sec latency 190.675 ms 2024-09-13T15:46:05.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1531862 11.86 MB/sec execute 581 sec latency 372.854 ms 2024-09-13T15:46:06.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 518447 4.00 MB/sec execute 582 sec latency 125.318 ms 2024-09-13T15:46:06.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1531937 11.84 MB/sec execute 582 sec latency 364.540 ms 2024-09-13T15:46:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:05 smithi090 ceph-mon[27811]: pgmap v623: 129 pgs: 129 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-13T15:46:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:05 smithi175 ceph-mon[36128]: pgmap v623: 129 pgs: 129 active+clean; 200 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.1 KiB/s rd, 1.8 MiB/s wr, 96 op/s 2024-09-13T15:46:07.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 519117 4.00 MB/sec execute 583 sec latency 11.695 ms 2024-09-13T15:46:07.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1535416 11.85 MB/sec execute 583 sec latency 62.814 ms 2024-09-13T15:46:08.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 519837 4.00 MB/sec execute 584 sec latency 11.121 ms 2024-09-13T15:46:08.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1538901 11.86 MB/sec execute 584 sec latency 59.427 ms 2024-09-13T15:46:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:07 smithi090 ceph-mon[27811]: pgmap v624: 129 pgs: 129 active+clean; 196 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.4 MiB/s wr, 75 op/s 2024-09-13T15:46:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:07 smithi175 ceph-mon[36128]: pgmap v624: 129 pgs: 129 active+clean; 196 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 KiB/s rd, 1.4 MiB/s wr, 75 op/s 2024-09-13T15:46:09.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 520729 4.00 MB/sec execute 585 sec latency 11.472 ms 2024-09-13T15:46:09.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1539139 11.84 MB/sec execute 585 sec latency 432.250 ms 2024-09-13T15:46:10.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 521646 4.00 MB/sec execute 586 sec latency 11.785 ms 2024-09-13T15:46:10.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1542577 11.86 MB/sec execute 586 sec latency 27.818 ms 2024-09-13T15:46:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:09 smithi090 ceph-mon[27811]: pgmap v625: 129 pgs: 129 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-09-13T15:46:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:09 smithi175 ceph-mon[36128]: pgmap v625: 129 pgs: 129 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 1.6 MiB/s wr, 97 op/s 2024-09-13T15:46:11.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 522418 4.01 MB/sec execute 587 sec latency 13.897 ms 2024-09-13T15:46:11.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1546105 11.87 MB/sec execute 587 sec latency 34.368 ms 2024-09-13T15:46:12.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 522830 4.00 MB/sec execute 588 sec latency 206.869 ms 2024-09-13T15:46:12.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1546943 11.86 MB/sec execute 588 sec latency 412.198 ms 2024-09-13T15:46:12.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:11 smithi090 ceph-mon[27811]: pgmap v626: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-13T15:46:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:11 smithi175 ceph-mon[36128]: pgmap v626: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 74 op/s 2024-09-13T15:46:13.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 523501 4.00 MB/sec execute 589 sec latency 53.013 ms 2024-09-13T15:46:13.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1547197 11.84 MB/sec execute 589 sec latency 380.489 ms 2024-09-13T15:46:14.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 524268 4.00 MB/sec execute 590 sec latency 142.265 ms 2024-09-13T15:46:14.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1549540 11.84 MB/sec execute 590 sec latency 334.073 ms 2024-09-13T15:46:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:13 smithi090 ceph-mon[27811]: pgmap v627: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:46:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:13 smithi175 ceph-mon[36128]: pgmap v627: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.9 KiB/s rd, 1.6 MiB/s wr, 83 op/s 2024-09-13T15:46:15.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 525196 4.00 MB/sec execute 591 sec latency 10.387 ms 2024-09-13T15:46:15.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1553074 11.85 MB/sec execute 591 sec latency 32.864 ms 2024-09-13T15:46:16.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 525985 4.01 MB/sec execute 592 sec latency 11.287 ms 2024-09-13T15:46:16.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1553324 11.83 MB/sec execute 592 sec latency 392.660 ms 2024-09-13T15:46:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:15 smithi090 ceph-mon[27811]: pgmap v628: 129 pgs: 129 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 101 op/s 2024-09-13T15:46:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:15 smithi175 ceph-mon[36128]: pgmap v628: 129 pgs: 129 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.0 MiB/s wr, 101 op/s 2024-09-13T15:46:17.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 526555 4.00 MB/sec execute 593 sec latency 10.618 ms 2024-09-13T15:46:17.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1556684 11.84 MB/sec execute 593 sec latency 485.219 ms 2024-09-13T15:46:18.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 527371 4.00 MB/sec execute 594 sec latency 35.178 ms 2024-09-13T15:46:18.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1560176 11.86 MB/sec execute 594 sec latency 33.201 ms 2024-09-13T15:46:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:17 smithi090 ceph-mon[27811]: pgmap v629: 129 pgs: 129 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:46:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:17 smithi175 ceph-mon[36128]: pgmap v629: 129 pgs: 129 active+clean; 199 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.6 MiB/s wr, 81 op/s 2024-09-13T15:46:19.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 528201 4.00 MB/sec execute 595 sec latency 10.653 ms 2024-09-13T15:46:19.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:19.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1563714 11.87 MB/sec execute 595 sec latency 50.143 ms 2024-09-13T15:46:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:20.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 529090 4.01 MB/sec execute 596 sec latency 11.314 ms 2024-09-13T15:46:20.185 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:19 smithi175 ceph-mon[36128]: pgmap v630: 129 pgs: 129 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 102 op/s 2024-09-13T15:46:20.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1566232 11.87 MB/sec execute 596 sec latency 33.653 ms 2024-09-13T15:46:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:19 smithi090 ceph-mon[27811]: pgmap v630: 129 pgs: 129 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.9 MiB/s wr, 102 op/s 2024-09-13T15:46:21.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 529763 4.01 MB/sec execute 597 sec latency 9.229 ms 2024-09-13T15:46:21.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1569058 11.87 MB/sec execute 597 sec latency 33.285 ms 2024-09-13T15:46:22.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 530485 4.00 MB/sec execute 598 sec latency 10.556 ms 2024-09-13T15:46:22.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1570936 11.87 MB/sec execute 598 sec latency 245.687 ms 2024-09-13T15:46:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:21 smithi090 ceph-mon[27811]: pgmap v631: 129 pgs: 129 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 77 op/s 2024-09-13T15:46:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:21 smithi175 ceph-mon[36128]: pgmap v631: 129 pgs: 129 active+clean; 197 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.8 MiB/s wr, 77 op/s 2024-09-13T15:46:23.024 INFO:tasks.workunit.client.1.smithi175.stdout: 1 531155 4.00 MB/sec execute 599 sec latency 133.367 ms 2024-09-13T15:46:23.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 1571012 11.85 MB/sec execute 599 sec latency 361.188 ms 2024-09-13T15:46:24.025 INFO:tasks.workunit.client.1.smithi175.stdout: 1 cleanup 600 sec 2024-09-13T15:46:24.222 INFO:tasks.workunit.client.0.smithi090.stdout: 1 cleanup 600 sec 2024-09-13T15:46:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:23 smithi090 ceph-mon[27811]: pgmap v632: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-13T15:46:24.252 INFO:tasks.workunit.client.0.smithi090.stdout: 0 cleanup 600 sec 2024-09-13T15:46:24.252 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Operation Count AvgLat MaxLat 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: ---------------------------------------- 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: NTCreateX 226046 0.009 9.859 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Close 166051 0.001 0.031 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Rename 9572 0.023 0.818 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Unlink 45647 0.020 34.667 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Deltree 6 1.757 4.459 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Mkdir 3 0.003 0.003 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Qpathinfo 204880 0.004 0.774 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Qfileinfo 35914 0.001 0.065 2024-09-13T15:46:24.253 INFO:tasks.workunit.client.0.smithi090.stdout: Qfsinfo 37568 0.002 0.032 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: Sfileinfo 18418 0.006 0.893 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: Find 79218 0.015 0.116 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: WriteX 112759 0.014 54.838 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: ReadX 354330 0.002 10.766 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: LockX 736 0.003 0.021 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: UnlockX 736 0.001 0.006 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: Flush 15847 37.267 528.583 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-09-13T15:46:24.254 INFO:tasks.workunit.client.0.smithi090.stdout:Throughput 11.8543 MB/sec 1 clients 1 procs max_latency=528.590 ms 2024-09-13T15:46:24.255 INFO:tasks.workunit.client.0.smithi090.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-13T15:46:24.255 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-09-13T15:46:24.255 INFO:tasks.workunit.client.0.smithi090.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-13T15:46:24.255 INFO:tasks.workunit.client.0.smithi090.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-13T15:46:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:23 smithi175 ceph-mon[36128]: pgmap v632: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.9 MiB/s wr, 93 op/s 2024-09-13T15:46:24.289 INFO:tasks.workunit.client.1.smithi175.stdout: 0 cleanup 600 sec 2024-09-13T15:46:24.289 INFO:tasks.workunit.client.1.smithi175.stdout: 2024-09-13T15:46:24.289 INFO:tasks.workunit.client.1.smithi175.stdout: Operation Count AvgLat MaxLat 2024-09-13T15:46:24.289 INFO:tasks.workunit.client.1.smithi175.stdout: ---------------------------------------- 2024-09-13T15:46:24.289 INFO:tasks.workunit.client.1.smithi175.stdout: NTCreateX 76458 2.239 220.506 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Close 56203 0.041 1.974 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Rename 3237 6.570 143.083 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Unlink 15409 2.462 195.130 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Deltree 2 101.796 151.119 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Mkdir 1 0.219 0.219 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Qpathinfo 69325 1.301 198.108 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Qfileinfo 12153 0.064 0.697 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Qfsinfo 12682 0.280 5.899 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Sfileinfo 6237 0.537 3.532 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: Find 26783 9.312 264.907 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: WriteX 38136 0.145 51.113 2024-09-13T15:46:24.290 INFO:tasks.workunit.client.1.smithi175.stdout: ReadX 119707 0.003 0.444 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout: LockX 248 0.562 1.030 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout: UnlockX 248 0.409 0.708 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout: Flush 5353 2.378 144.408 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout: 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout:Throughput 4.00184 MB/sec 1 clients 1 procs max_latency=264.913 ms 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout: 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-13T15:46:24.291 INFO:tasks.workunit.client.1.smithi175.stdout:8 of 10 processes prepared for launch 0 sec 2024-09-13T15:46:24.354 INFO:tasks.workunit.client.0.smithi090.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-13T15:46:24.354 INFO:tasks.workunit.client.0.smithi090.stdout:releasing clients 2024-09-13T15:46:24.390 INFO:tasks.workunit.client.1.smithi175.stdout:10 of 10 processes prepared for launch 0 sec 2024-09-13T15:46:24.391 INFO:tasks.workunit.client.1.smithi175.stdout:releasing clients 2024-09-13T15:46:25.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 664 211.59 MB/sec warmup 1 sec latency 617.657 ms 2024-09-13T15:46:25.390 INFO:tasks.workunit.client.1.smithi175.stdout: 10 172 70.37 MB/sec warmup 1 sec latency 495.761 ms 2024-09-13T15:46:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:25 smithi090 ceph-mon[27811]: pgmap v633: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:46:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:25 smithi175 ceph-mon[36128]: pgmap v633: 129 pgs: 129 active+clean; 198 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.1 MiB/s wr, 102 op/s 2024-09-13T15:46:26.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 730 106.56 MB/sec warmup 2 sec latency 612.350 ms 2024-09-13T15:46:26.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 460 80.67 MB/sec warmup 2 sec latency 27.184 ms 2024-09-13T15:46:27.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 913 72.56 MB/sec warmup 3 sec latency 206.623 ms 2024-09-13T15:46:27.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 709 70.83 MB/sec warmup 3 sec latency 89.322 ms 2024-09-13T15:46:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:27 smithi090 ceph-mon[27811]: pgmap v634: 129 pgs: 129 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6.4 MiB/s wr, 96 op/s 2024-09-13T15:46:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:27 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:46:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:27 smithi175 ceph-mon[36128]: pgmap v634: 129 pgs: 129 active+clean; 251 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 6.4 MiB/s wr, 96 op/s 2024-09-13T15:46:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:27 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:46:28.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 980 54.75 MB/sec warmup 4 sec latency 552.525 ms 2024-09-13T15:46:28.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 846 53.87 MB/sec warmup 4 sec latency 243.078 ms 2024-09-13T15:46:29.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 1513 52.14 MB/sec warmup 5 sec latency 470.881 ms 2024-09-13T15:46:29.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 990 44.52 MB/sec warmup 5 sec latency 107.440 ms 2024-09-13T15:46:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:29 smithi090 ceph-mon[27811]: pgmap v635: 129 pgs: 129 active+clean; 346 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 15 MiB/s wr, 173 op/s 2024-09-13T15:46:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:29 smithi175 ceph-mon[36128]: pgmap v635: 129 pgs: 129 active+clean; 346 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 15 MiB/s wr, 173 op/s 2024-09-13T15:46:30.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 1684 43.69 MB/sec warmup 6 sec latency 682.948 ms 2024-09-13T15:46:30.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 1125 38.29 MB/sec warmup 6 sec latency 336.296 ms 2024-09-13T15:46:31.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 1886 37.68 MB/sec warmup 7 sec latency 485.761 ms 2024-09-13T15:46:31.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 1256 33.02 MB/sec warmup 7 sec latency 164.266 ms 2024-09-13T15:46:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:31 smithi090 ceph-mon[27811]: pgmap v636: 129 pgs: 129 active+clean; 363 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 152 op/s 2024-09-13T15:46:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:31 smithi175 ceph-mon[36128]: pgmap v636: 129 pgs: 129 active+clean; 363 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 17 MiB/s wr, 152 op/s 2024-09-13T15:46:32.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 2388 35.00 MB/sec warmup 8 sec latency 414.786 ms 2024-09-13T15:46:32.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 1425 32.51 MB/sec warmup 8 sec latency 257.000 ms 2024-09-13T15:46:33.354 INFO:tasks.workunit.client.0.smithi090.stdout: 10 2565 32.99 MB/sec warmup 9 sec latency 466.745 ms 2024-09-13T15:46:33.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 1556 29.02 MB/sec warmup 9 sec latency 113.339 ms 2024-09-13T15:46:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:33 smithi090 ceph-mon[27811]: pgmap v637: 129 pgs: 129 active+clean; 375 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 19 MiB/s wr, 214 op/s 2024-09-13T15:46:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:33 smithi175 ceph-mon[36128]: pgmap v637: 129 pgs: 129 active+clean; 375 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 19 MiB/s wr, 214 op/s 2024-09-13T15:46:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:34.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 2986 33.52 MB/sec warmup 10 sec latency 457.122 ms 2024-09-13T15:46:34.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 1740 26.26 MB/sec warmup 10 sec latency 101.845 ms 2024-09-13T15:46:35.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 3150 31.06 MB/sec warmup 11 sec latency 517.414 ms 2024-09-13T15:46:35.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 1887 24.00 MB/sec warmup 11 sec latency 183.568 ms 2024-09-13T15:46:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:35 smithi090 ceph-mon[27811]: pgmap v638: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 19 MiB/s wr, 211 op/s 2024-09-13T15:46:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:35 smithi175 ceph-mon[36128]: pgmap v638: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.9 KiB/s rd, 19 MiB/s wr, 211 op/s 2024-09-13T15:46:36.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 3308 29.47 MB/sec warmup 12 sec latency 713.380 ms 2024-09-13T15:46:36.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 2072 22.19 MB/sec warmup 12 sec latency 192.718 ms 2024-09-13T15:46:37.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 3967 29.85 MB/sec warmup 13 sec latency 473.451 ms 2024-09-13T15:46:37.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 2269 21.13 MB/sec warmup 13 sec latency 123.614 ms 2024-09-13T15:46:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:37 smithi090 ceph-mon[27811]: pgmap v639: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 19 MiB/s wr, 207 op/s 2024-09-13T15:46:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:37 smithi175 ceph-mon[36128]: pgmap v639: 129 pgs: 129 active+clean; 375 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 19 MiB/s wr, 207 op/s 2024-09-13T15:46:38.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4240 28.78 MB/sec warmup 14 sec latency 592.681 ms 2024-09-13T15:46:38.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 2384 19.97 MB/sec warmup 14 sec latency 121.282 ms 2024-09-13T15:46:39.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4245 26.86 MB/sec warmup 15 sec latency 560.340 ms 2024-09-13T15:46:39.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 2523 19.21 MB/sec warmup 15 sec latency 99.069 ms 2024-09-13T15:46:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:39 smithi090 ceph-mon[27811]: pgmap v640: 129 pgs: 129 active+clean; 378 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 14 MiB/s wr, 237 op/s 2024-09-13T15:46:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:39 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:46:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:39 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:46:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:39 smithi175 ceph-mon[36128]: pgmap v640: 129 pgs: 129 active+clean; 378 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 14 MiB/s wr, 237 op/s 2024-09-13T15:46:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:39 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:46:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:39 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:46:40.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4289 25.25 MB/sec warmup 16 sec latency 431.414 ms 2024-09-13T15:46:40.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 2691 19.09 MB/sec warmup 16 sec latency 117.647 ms 2024-09-13T15:46:41.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:46:41.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:46:41.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:46:41.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:46:41.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:46:41.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:46:41.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4298 23.78 MB/sec warmup 17 sec latency 615.622 ms 2024-09-13T15:46:41.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 2954 19.67 MB/sec warmup 17 sec latency 135.187 ms 2024-09-13T15:46:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:41 smithi090 ceph-mon[27811]: pgmap v641: 129 pgs: 129 active+clean; 378 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 5.2 MiB/s wr, 161 op/s 2024-09-13T15:46:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:41 smithi175 ceph-mon[36128]: pgmap v641: 129 pgs: 129 active+clean; 378 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 5.2 MiB/s wr, 161 op/s 2024-09-13T15:46:42.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4301 22.46 MB/sec warmup 18 sec latency 567.314 ms 2024-09-13T15:46:42.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3228 19.32 MB/sec warmup 18 sec latency 175.550 ms 2024-09-13T15:46:43.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4343 21.33 MB/sec warmup 19 sec latency 648.826 ms 2024-09-13T15:46:43.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3293 18.60 MB/sec warmup 19 sec latency 208.603 ms 2024-09-13T15:46:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:43 smithi090 ceph-mon[27811]: pgmap v642: 129 pgs: 129 active+clean; 386 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.7 MiB/s wr, 174 op/s 2024-09-13T15:46:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:43 smithi175 ceph-mon[36128]: pgmap v642: 129 pgs: 129 active+clean; 386 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.7 MiB/s wr, 174 op/s 2024-09-13T15:46:44.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4362 20.29 MB/sec warmup 20 sec latency 832.873 ms 2024-09-13T15:46:44.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3320 17.68 MB/sec warmup 20 sec latency 109.911 ms 2024-09-13T15:46:45.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4367 19.33 MB/sec warmup 21 sec latency 626.825 ms 2024-09-13T15:46:45.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3375 17.01 MB/sec warmup 21 sec latency 155.488 ms 2024-09-13T15:46:46.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:45 smithi175 ceph-mon[36128]: pgmap v643: 129 pgs: 129 active+clean; 398 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 152 op/s 2024-09-13T15:46:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:45 smithi090 ceph-mon[27811]: pgmap v643: 129 pgs: 129 active+clean; 398 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 152 op/s 2024-09-13T15:46:46.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4378 18.46 MB/sec warmup 22 sec latency 578.849 ms 2024-09-13T15:46:46.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3570 17.48 MB/sec warmup 22 sec latency 168.080 ms 2024-09-13T15:46:47.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4443 17.73 MB/sec warmup 23 sec latency 494.054 ms 2024-09-13T15:46:47.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3726 17.04 MB/sec warmup 23 sec latency 179.205 ms 2024-09-13T15:46:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:47 smithi090 ceph-mon[27811]: pgmap v644: 129 pgs: 129 active+clean; 398 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.7 MiB/s wr, 144 op/s 2024-09-13T15:46:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:47 smithi175 ceph-mon[36128]: pgmap v644: 129 pgs: 129 active+clean; 398 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.7 MiB/s wr, 144 op/s 2024-09-13T15:46:48.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4487 17.04 MB/sec warmup 24 sec latency 548.656 ms 2024-09-13T15:46:48.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3830 16.57 MB/sec warmup 24 sec latency 131.749 ms 2024-09-13T15:46:49.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:46:49.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4538 16.40 MB/sec warmup 25 sec latency 683.837 ms 2024-09-13T15:46:49.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3924 16.03 MB/sec warmup 25 sec latency 114.404 ms 2024-09-13T15:46:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:49 smithi090 ceph-mon[27811]: pgmap v645: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.4 MiB/s wr, 165 op/s 2024-09-13T15:46:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:49 smithi175 ceph-mon[36128]: pgmap v645: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.8 KiB/s rd, 4.4 MiB/s wr, 165 op/s 2024-09-13T15:46:50.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 4978 17.06 MB/sec warmup 26 sec latency 712.464 ms 2024-09-13T15:46:50.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 3999 15.41 MB/sec warmup 26 sec latency 136.084 ms 2024-09-13T15:46:51.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 5248 16.79 MB/sec warmup 27 sec latency 688.603 ms 2024-09-13T15:46:51.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4075 14.84 MB/sec warmup 27 sec latency 151.612 ms 2024-09-13T15:46:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:51 smithi090 ceph-mon[27811]: pgmap v646: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.4 MiB/s wr, 115 op/s 2024-09-13T15:46:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:51 smithi175 ceph-mon[36128]: pgmap v646: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.4 MiB/s wr, 115 op/s 2024-09-13T15:46:52.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 5450 16.25 MB/sec warmup 28 sec latency 536.641 ms 2024-09-13T15:46:52.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4179 14.33 MB/sec warmup 28 sec latency 207.814 ms 2024-09-13T15:46:53.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 5769 16.05 MB/sec warmup 29 sec latency 529.075 ms 2024-09-13T15:46:53.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4310 13.94 MB/sec warmup 29 sec latency 153.561 ms 2024-09-13T15:46:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:53 smithi090 ceph-mon[27811]: pgmap v647: 129 pgs: 129 active+clean; 399 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.6 MiB/s wr, 113 op/s 2024-09-13T15:46:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:53 smithi175 ceph-mon[36128]: pgmap v647: 129 pgs: 129 active+clean; 399 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.6 MiB/s wr, 113 op/s 2024-09-13T15:46:54.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 5964 15.73 MB/sec warmup 30 sec latency 712.867 ms 2024-09-13T15:46:54.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4444 13.58 MB/sec warmup 30 sec latency 190.753 ms 2024-09-13T15:46:55.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 6237 16.16 MB/sec warmup 31 sec latency 641.060 ms 2024-09-13T15:46:55.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4572 13.32 MB/sec warmup 31 sec latency 159.484 ms 2024-09-13T15:46:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:55 smithi090 ceph-mon[27811]: pgmap v648: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.8 MiB/s wr, 136 op/s 2024-09-13T15:46:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:55 smithi175 ceph-mon[36128]: pgmap v648: 129 pgs: 129 active+clean; 397 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.8 MiB/s wr, 136 op/s 2024-09-13T15:46:56.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 6437 16.30 MB/sec warmup 32 sec latency 824.268 ms 2024-09-13T15:46:56.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4722 13.17 MB/sec warmup 32 sec latency 139.924 ms 2024-09-13T15:46:57.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 7781 18.00 MB/sec warmup 33 sec latency 636.402 ms 2024-09-13T15:46:57.391 INFO:tasks.workunit.client.1.smithi175.stdout: 10 4907 13.18 MB/sec warmup 33 sec latency 90.708 ms 2024-09-13T15:46:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:57 smithi090 ceph-mon[27811]: pgmap v649: 129 pgs: 129 active+clean; 390 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.0 MiB/s wr, 117 op/s 2024-09-13T15:46:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:57 smithi175 ceph-mon[36128]: pgmap v649: 129 pgs: 129 active+clean; 390 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.0 MiB/s wr, 117 op/s 2024-09-13T15:46:58.355 INFO:tasks.workunit.client.0.smithi090.stdout: 10 7891 17.55 MB/sec warmup 34 sec latency 140.705 ms 2024-09-13T15:46:58.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 5104 13.30 MB/sec warmup 34 sec latency 67.382 ms 2024-09-13T15:46:59.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 8092 17.18 MB/sec warmup 35 sec latency 117.300 ms 2024-09-13T15:46:59.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 5291 12.96 MB/sec warmup 35 sec latency 67.655 ms 2024-09-13T15:47:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:46:59 smithi090 ceph-mon[27811]: pgmap v650: 129 pgs: 129 active+clean; 379 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.5 MiB/s wr, 193 op/s 2024-09-13T15:47:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:46:59 smithi175 ceph-mon[36128]: pgmap v650: 129 pgs: 129 active+clean; 379 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.5 MiB/s wr, 193 op/s 2024-09-13T15:47:00.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 9518 18.44 MB/sec warmup 36 sec latency 125.016 ms 2024-09-13T15:47:00.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 5497 12.66 MB/sec warmup 36 sec latency 51.291 ms 2024-09-13T15:47:01.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 10115 19.42 MB/sec warmup 37 sec latency 421.121 ms 2024-09-13T15:47:01.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 5693 12.41 MB/sec warmup 37 sec latency 67.122 ms 2024-09-13T15:47:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:01 smithi090 ceph-mon[27811]: pgmap v651: 129 pgs: 129 active+clean; 379 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.0 MiB/s wr, 161 op/s 2024-09-13T15:47:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:01 smithi175 ceph-mon[36128]: pgmap v651: 129 pgs: 129 active+clean; 379 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.0 MiB/s wr, 161 op/s 2024-09-13T15:47:02.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 10348 19.23 MB/sec warmup 38 sec latency 598.459 ms 2024-09-13T15:47:02.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 5870 12.30 MB/sec warmup 38 sec latency 173.490 ms 2024-09-13T15:47:03.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 10416 18.87 MB/sec warmup 39 sec latency 644.093 ms 2024-09-13T15:47:03.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 5972 12.11 MB/sec warmup 39 sec latency 122.894 ms 2024-09-13T15:47:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:03 smithi090 ceph-mon[27811]: pgmap v652: 129 pgs: 129 active+clean; 378 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 2.7 MiB/s wr, 192 op/s 2024-09-13T15:47:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:03 smithi175 ceph-mon[36128]: pgmap v652: 129 pgs: 129 active+clean; 378 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 2.7 MiB/s wr, 192 op/s 2024-09-13T15:47:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:04.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 10881 19.02 MB/sec warmup 40 sec latency 551.256 ms 2024-09-13T15:47:04.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6090 11.99 MB/sec warmup 40 sec latency 142.162 ms 2024-09-13T15:47:05.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11327 19.15 MB/sec warmup 41 sec latency 722.239 ms 2024-09-13T15:47:05.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6307 12.26 MB/sec warmup 41 sec latency 84.418 ms 2024-09-13T15:47:06.185 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:05 smithi175 ceph-mon[36128]: pgmap v653: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 2.9 MiB/s wr, 238 op/s 2024-09-13T15:47:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:05 smithi090 ceph-mon[27811]: pgmap v653: 129 pgs: 129 active+clean; 381 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 9.8 KiB/s rd, 2.9 MiB/s wr, 238 op/s 2024-09-13T15:47:06.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11332 18.70 MB/sec warmup 42 sec latency 764.730 ms 2024-09-13T15:47:06.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6575 12.59 MB/sec warmup 42 sec latency 164.520 ms 2024-09-13T15:47:07.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11358 18.28 MB/sec warmup 43 sec latency 546.244 ms 2024-09-13T15:47:07.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6821 12.61 MB/sec warmup 43 sec latency 129.245 ms 2024-09-13T15:47:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:07 smithi090 ceph-mon[27811]: pgmap v654: 129 pgs: 129 active+clean; 392 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-09-13T15:47:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:07 smithi175 ceph-mon[36128]: pgmap v654: 129 pgs: 129 active+clean; 392 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.5 MiB/s wr, 229 op/s 2024-09-13T15:47:08.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11383 17.88 MB/sec warmup 44 sec latency 473.099 ms 2024-09-13T15:47:08.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6858 12.38 MB/sec warmup 44 sec latency 142.456 ms 2024-09-13T15:47:09.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11388 17.48 MB/sec warmup 45 sec latency 493.893 ms 2024-09-13T15:47:09.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6883 12.11 MB/sec warmup 45 sec latency 114.074 ms 2024-09-13T15:47:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:09 smithi090 ceph-mon[27811]: pgmap v655: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.2 MiB/s wr, 224 op/s 2024-09-13T15:47:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:09 smithi175 ceph-mon[36128]: pgmap v655: 129 pgs: 129 active+clean; 402 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.2 MiB/s wr, 224 op/s 2024-09-13T15:47:10.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11420 17.12 MB/sec warmup 46 sec latency 503.493 ms 2024-09-13T15:47:10.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 6972 12.05 MB/sec warmup 46 sec latency 123.286 ms 2024-09-13T15:47:11.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11451 16.77 MB/sec warmup 47 sec latency 528.692 ms 2024-09-13T15:47:11.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7179 12.32 MB/sec warmup 47 sec latency 150.382 ms 2024-09-13T15:47:11.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:11 smithi090 ceph-mon[27811]: pgmap v656: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-09-13T15:47:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:11 smithi175 ceph-mon[36128]: pgmap v656: 129 pgs: 129 active+clean; 400 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.5 MiB/s wr, 143 op/s 2024-09-13T15:47:12.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11456 16.42 MB/sec warmup 48 sec latency 591.981 ms 2024-09-13T15:47:12.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7293 12.16 MB/sec warmup 48 sec latency 223.974 ms 2024-09-13T15:47:13.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 11466 16.09 MB/sec warmup 49 sec latency 599.395 ms 2024-09-13T15:47:13.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7397 12.03 MB/sec warmup 49 sec latency 151.787 ms 2024-09-13T15:47:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:13 smithi090 ceph-mon[27811]: pgmap v657: 129 pgs: 129 active+clean; 404 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 184 op/s 2024-09-13T15:47:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:13 smithi175 ceph-mon[36128]: pgmap v657: 129 pgs: 129 active+clean; 404 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.2 MiB/s wr, 184 op/s 2024-09-13T15:47:14.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 12415 16.73 MB/sec warmup 50 sec latency 477.030 ms 2024-09-13T15:47:14.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7503 11.84 MB/sec warmup 50 sec latency 129.157 ms 2024-09-13T15:47:15.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 13556 17.73 MB/sec warmup 51 sec latency 144.294 ms 2024-09-13T15:47:15.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7618 11.61 MB/sec warmup 51 sec latency 72.114 ms 2024-09-13T15:47:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:15 smithi090 ceph-mon[27811]: pgmap v658: 129 pgs: 129 active+clean; 405 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.8 MiB/s wr, 159 op/s 2024-09-13T15:47:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:15 smithi175 ceph-mon[36128]: pgmap v658: 129 pgs: 129 active+clean; 405 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.8 MiB/s wr, 159 op/s 2024-09-13T15:47:16.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 14877 18.78 MB/sec warmup 52 sec latency 121.300 ms 2024-09-13T15:47:16.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7759 11.41 MB/sec warmup 52 sec latency 100.245 ms 2024-09-13T15:47:17.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 14967 18.47 MB/sec warmup 53 sec latency 239.396 ms 2024-09-13T15:47:17.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 7919 11.27 MB/sec warmup 53 sec latency 103.055 ms 2024-09-13T15:47:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:17 smithi090 ceph-mon[27811]: pgmap v659: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 112 op/s 2024-09-13T15:47:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:17 smithi175 ceph-mon[36128]: pgmap v659: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 112 op/s 2024-09-13T15:47:18.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 15152 18.21 MB/sec warmup 54 sec latency 234.984 ms 2024-09-13T15:47:18.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 8071 11.16 MB/sec warmup 54 sec latency 105.152 ms 2024-09-13T15:47:19.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:19.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 15814 18.66 MB/sec warmup 55 sec latency 677.866 ms 2024-09-13T15:47:19.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 8227 11.13 MB/sec warmup 55 sec latency 109.869 ms 2024-09-13T15:47:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:19 smithi090 ceph-mon[27811]: pgmap v660: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 4.2 MiB/s wr, 149 op/s 2024-09-13T15:47:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:19 smithi175 ceph-mon[36128]: pgmap v660: 129 pgs: 129 active+clean; 406 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 4.2 MiB/s wr, 149 op/s 2024-09-13T15:47:20.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16011 18.36 MB/sec warmup 56 sec latency 742.409 ms 2024-09-13T15:47:20.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 8380 11.06 MB/sec warmup 56 sec latency 107.335 ms 2024-09-13T15:47:21.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16314 18.18 MB/sec warmup 57 sec latency 772.765 ms 2024-09-13T15:47:21.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 8572 11.28 MB/sec warmup 57 sec latency 167.674 ms 2024-09-13T15:47:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:21 smithi090 ceph-mon[27811]: pgmap v661: 129 pgs: 129 active+clean; 404 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.1 MiB/s wr, 138 op/s 2024-09-13T15:47:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:21 smithi175 ceph-mon[36128]: pgmap v661: 129 pgs: 129 active+clean; 404 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.1 MiB/s wr, 138 op/s 2024-09-13T15:47:22.356 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16579 18.01 MB/sec warmup 58 sec latency 681.451 ms 2024-09-13T15:47:22.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 8700 11.10 MB/sec warmup 58 sec latency 164.462 ms 2024-09-13T15:47:23.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 16669 17.84 MB/sec warmup 59 sec latency 868.361 ms 2024-09-13T15:47:23.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 8854 10.93 MB/sec warmup 59 sec latency 152.045 ms 2024-09-13T15:47:24.084 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:23 smithi090 ceph-mon[27811]: pgmap v662: 129 pgs: 129 active+clean; 390 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-09-13T15:47:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:23 smithi175 ceph-mon[36128]: pgmap v662: 129 pgs: 129 active+clean; 390 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.7 MiB/s wr, 199 op/s 2024-09-13T15:47:24.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 17001 18.12 MB/sec warmup 60 sec latency 712.654 ms 2024-09-13T15:47:24.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9014 10.78 MB/sec warmup 60 sec latency 142.012 ms 2024-09-13T15:47:25.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 17158 18.03 MB/sec warmup 61 sec latency 569.039 ms 2024-09-13T15:47:25.392 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9194 10.64 MB/sec warmup 61 sec latency 111.052 ms 2024-09-13T15:47:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:25 smithi090 ceph-mon[27811]: pgmap v663: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-09-13T15:47:26.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:25 smithi175 ceph-mon[36128]: pgmap v663: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.3 MiB/s wr, 192 op/s 2024-09-13T15:47:26.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 17320 17.83 MB/sec warmup 62 sec latency 703.323 ms 2024-09-13T15:47:26.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9342 10.55 MB/sec warmup 62 sec latency 173.971 ms 2024-09-13T15:47:27.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 17496 17.75 MB/sec warmup 63 sec latency 523.400 ms 2024-09-13T15:47:27.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9478 10.51 MB/sec warmup 63 sec latency 134.498 ms 2024-09-13T15:47:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:27 smithi090 ceph-mon[27811]: pgmap v664: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-09-13T15:47:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:27 smithi175 ceph-mon[36128]: pgmap v664: 129 pgs: 129 active+clean; 382 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-09-13T15:47:28.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 17608 17.55 MB/sec warmup 64 sec latency 669.310 ms 2024-09-13T15:47:28.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9568 10.41 MB/sec warmup 64 sec latency 318.259 ms 2024-09-13T15:47:29.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 18065 17.66 MB/sec warmup 65 sec latency 567.653 ms 2024-09-13T15:47:29.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9727 10.43 MB/sec warmup 65 sec latency 169.785 ms 2024-09-13T15:47:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:29 smithi090 ceph-mon[27811]: pgmap v665: 129 pgs: 129 active+clean; 385 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.1 MiB/s wr, 239 op/s 2024-09-13T15:47:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:29 smithi175 ceph-mon[36128]: pgmap v665: 129 pgs: 129 active+clean; 385 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 3.1 MiB/s wr, 239 op/s 2024-09-13T15:47:30.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 18423 17.69 MB/sec warmup 66 sec latency 731.412 ms 2024-09-13T15:47:30.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 9995 10.76 MB/sec warmup 66 sec latency 118.857 ms 2024-09-13T15:47:31.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 18427 17.43 MB/sec warmup 67 sec latency 746.966 ms 2024-09-13T15:47:31.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 10196 10.80 MB/sec warmup 67 sec latency 165.240 ms 2024-09-13T15:47:32.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:31 smithi175 ceph-mon[36128]: pgmap v666: 129 pgs: 129 active+clean; 386 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-09-13T15:47:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:31 smithi090 ceph-mon[27811]: pgmap v666: 129 pgs: 129 active+clean; 386 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-09-13T15:47:32.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 18491 17.19 MB/sec warmup 68 sec latency 484.794 ms 2024-09-13T15:47:32.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 10391 10.82 MB/sec warmup 68 sec latency 169.355 ms 2024-09-13T15:47:33.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 18602 16.99 MB/sec warmup 69 sec latency 348.270 ms 2024-09-13T15:47:33.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 10419 10.67 MB/sec warmup 69 sec latency 91.589 ms 2024-09-13T15:47:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:33 smithi090 ceph-mon[27811]: pgmap v667: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-09-13T15:47:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:33 smithi175 ceph-mon[36128]: pgmap v667: 129 pgs: 129 active+clean; 403 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 4.1 MiB/s wr, 195 op/s 2024-09-13T15:47:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:34.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 19363 17.39 MB/sec warmup 70 sec latency 449.081 ms 2024-09-13T15:47:34.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 10515 10.65 MB/sec warmup 70 sec latency 71.317 ms 2024-09-13T15:47:35.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 19690 17.20 MB/sec warmup 71 sec latency 558.493 ms 2024-09-13T15:47:35.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 10741 10.86 MB/sec warmup 71 sec latency 93.966 ms 2024-09-13T15:47:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:35 smithi090 ceph-mon[27811]: pgmap v668: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 160 op/s 2024-09-13T15:47:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:35 smithi175 ceph-mon[36128]: pgmap v668: 129 pgs: 129 active+clean; 408 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.9 MiB/s wr, 160 op/s 2024-09-13T15:47:36.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 20978 18.10 MB/sec warmup 72 sec latency 124.402 ms 2024-09-13T15:47:36.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 10903 10.82 MB/sec warmup 72 sec latency 97.216 ms 2024-09-13T15:47:37.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22008 18.62 MB/sec warmup 73 sec latency 279.461 ms 2024-09-13T15:47:37.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11005 10.71 MB/sec warmup 73 sec latency 98.293 ms 2024-09-13T15:47:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:37 smithi090 ceph-mon[27811]: pgmap v669: 129 pgs: 129 active+clean; 410 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.0 MiB/s wr, 129 op/s 2024-09-13T15:47:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:37 smithi175 ceph-mon[36128]: pgmap v669: 129 pgs: 129 active+clean; 410 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.0 MiB/s wr, 129 op/s 2024-09-13T15:47:38.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22016 18.37 MB/sec warmup 74 sec latency 539.541 ms 2024-09-13T15:47:38.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11069 10.58 MB/sec warmup 74 sec latency 316.934 ms 2024-09-13T15:47:39.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22045 18.14 MB/sec warmup 75 sec latency 772.116 ms 2024-09-13T15:47:39.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11146 10.45 MB/sec warmup 75 sec latency 440.320 ms 2024-09-13T15:47:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:39 smithi090 ceph-mon[27811]: pgmap v670: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.8 MiB/s wr, 165 op/s 2024-09-13T15:47:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:39 smithi175 ceph-mon[36128]: pgmap v670: 129 pgs: 129 active+clean; 411 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.8 MiB/s wr, 165 op/s 2024-09-13T15:47:40.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22062 17.90 MB/sec warmup 76 sec latency 579.106 ms 2024-09-13T15:47:40.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11249 10.32 MB/sec warmup 76 sec latency 134.613 ms 2024-09-13T15:47:41.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:40 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:47:41.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:40 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:47:41.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22072 17.68 MB/sec warmup 77 sec latency 597.640 ms 2024-09-13T15:47:41.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11387 10.21 MB/sec warmup 77 sec latency 131.474 ms 2024-09-13T15:47:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:41 smithi090 ceph-mon[27811]: pgmap v671: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.6 MiB/s wr, 124 op/s 2024-09-13T15:47:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:41 smithi175 ceph-mon[36128]: pgmap v671: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.6 MiB/s wr, 124 op/s 2024-09-13T15:47:42.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22126 17.47 MB/sec warmup 78 sec latency 510.764 ms 2024-09-13T15:47:42.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11529 10.15 MB/sec warmup 78 sec latency 124.645 ms 2024-09-13T15:47:43.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22132 17.25 MB/sec warmup 79 sec latency 376.021 ms 2024-09-13T15:47:43.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11660 10.08 MB/sec warmup 79 sec latency 143.557 ms 2024-09-13T15:47:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:43 smithi090 ceph-mon[27811]: pgmap v672: 129 pgs: 129 active+clean; 401 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.6 MiB/s wr, 136 op/s 2024-09-13T15:47:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:47:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:47:44.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:47:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:43 smithi175 ceph-mon[36128]: pgmap v672: 129 pgs: 129 active+clean; 401 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.6 MiB/s wr, 136 op/s 2024-09-13T15:47:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:47:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:47:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:47:44.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22144 17.03 MB/sec warmup 80 sec latency 554.549 ms 2024-09-13T15:47:44.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11790 10.11 MB/sec warmup 80 sec latency 237.251 ms 2024-09-13T15:47:45.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22191 16.84 MB/sec warmup 81 sec latency 706.682 ms 2024-09-13T15:47:45.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 11957 10.07 MB/sec warmup 81 sec latency 279.696 ms 2024-09-13T15:47:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:45 smithi090 ceph-mon[27811]: pgmap v673: 129 pgs: 129 active+clean; 399 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-09-13T15:47:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:45 smithi175 ceph-mon[36128]: pgmap v673: 129 pgs: 129 active+clean; 399 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.5 MiB/s wr, 169 op/s 2024-09-13T15:47:46.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22244 16.65 MB/sec warmup 82 sec latency 706.819 ms 2024-09-13T15:47:46.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 12155 10.17 MB/sec warmup 82 sec latency 80.596 ms 2024-09-13T15:47:47.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22346 16.51 MB/sec warmup 83 sec latency 604.273 ms 2024-09-13T15:47:47.393 INFO:tasks.workunit.client.1.smithi175.stdout: 10 12298 10.07 MB/sec warmup 83 sec latency 123.151 ms 2024-09-13T15:47:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:47 smithi090 ceph-mon[27811]: pgmap v674: 129 pgs: 129 active+clean; 392 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.4 MiB/s wr, 164 op/s 2024-09-13T15:47:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:47 smithi175 ceph-mon[36128]: pgmap v674: 129 pgs: 129 active+clean; 392 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.4 MiB/s wr, 164 op/s 2024-09-13T15:47:48.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22678 16.62 MB/sec warmup 84 sec latency 661.738 ms 2024-09-13T15:47:48.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 12458 9.96 MB/sec warmup 84 sec latency 143.159 ms 2024-09-13T15:47:49.167 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:47:49.357 INFO:tasks.workunit.client.0.smithi090.stdout: 10 22943 16.58 MB/sec warmup 85 sec latency 751.952 ms 2024-09-13T15:47:49.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 12620 9.86 MB/sec warmup 85 sec latency 172.684 ms 2024-09-13T15:47:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:49 smithi090 ceph-mon[27811]: pgmap v675: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-09-13T15:47:50.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:49 smithi175 ceph-mon[36128]: pgmap v675: 129 pgs: 129 active+clean; 389 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-09-13T15:47:50.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 23139 16.41 MB/sec warmup 86 sec latency 785.186 ms 2024-09-13T15:47:50.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 12779 9.79 MB/sec warmup 86 sec latency 207.699 ms 2024-09-13T15:47:51.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 23261 16.24 MB/sec warmup 87 sec latency 596.844 ms 2024-09-13T15:47:51.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 12926 9.73 MB/sec warmup 87 sec latency 133.820 ms 2024-09-13T15:47:52.190 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:51 smithi175 ceph-mon[36128]: pgmap v676: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.5 MiB/s wr, 176 op/s 2024-09-13T15:47:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:51 smithi090 ceph-mon[27811]: pgmap v676: 129 pgs: 129 active+clean; 389 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.5 MiB/s wr, 176 op/s 2024-09-13T15:47:52.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 23557 16.17 MB/sec warmup 88 sec latency 823.699 ms 2024-09-13T15:47:52.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 13072 9.72 MB/sec warmup 88 sec latency 137.368 ms 2024-09-13T15:47:53.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 23789 16.12 MB/sec warmup 89 sec latency 720.430 ms 2024-09-13T15:47:53.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 13224 9.74 MB/sec warmup 89 sec latency 128.871 ms 2024-09-13T15:47:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:53 smithi090 ceph-mon[27811]: pgmap v677: 129 pgs: 129 active+clean; 391 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-13T15:47:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:53 smithi175 ceph-mon[36128]: pgmap v677: 129 pgs: 129 active+clean; 391 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.5 MiB/s wr, 200 op/s 2024-09-13T15:47:54.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 24020 16.21 MB/sec warmup 90 sec latency 604.483 ms 2024-09-13T15:47:54.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 13410 9.79 MB/sec warmup 90 sec latency 147.963 ms 2024-09-13T15:47:55.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 24236 16.25 MB/sec warmup 91 sec latency 760.975 ms 2024-09-13T15:47:55.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 13674 10.00 MB/sec warmup 91 sec latency 99.628 ms 2024-09-13T15:47:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:55 smithi090 ceph-mon[27811]: pgmap v678: 129 pgs: 129 active+clean; 390 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-09-13T15:47:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:55 smithi175 ceph-mon[36128]: pgmap v678: 129 pgs: 129 active+clean; 390 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.1 KiB/s rd, 2.9 MiB/s wr, 224 op/s 2024-09-13T15:47:56.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 24445 16.18 MB/sec warmup 92 sec latency 681.751 ms 2024-09-13T15:47:56.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 13849 9.99 MB/sec warmup 92 sec latency 143.239 ms 2024-09-13T15:47:57.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 24597 16.11 MB/sec warmup 93 sec latency 599.879 ms 2024-09-13T15:47:57.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 13954 9.97 MB/sec warmup 93 sec latency 169.242 ms 2024-09-13T15:47:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:57 smithi090 ceph-mon[27811]: pgmap v679: 129 pgs: 129 active+clean; 396 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.1 MiB/s wr, 174 op/s 2024-09-13T15:47:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:57 smithi175 ceph-mon[36128]: pgmap v679: 129 pgs: 129 active+clean; 396 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.1 MiB/s wr, 174 op/s 2024-09-13T15:47:58.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 24733 16.03 MB/sec warmup 94 sec latency 608.565 ms 2024-09-13T15:47:58.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14020 9.93 MB/sec warmup 94 sec latency 98.869 ms 2024-09-13T15:47:59.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 25285 16.17 MB/sec warmup 95 sec latency 689.035 ms 2024-09-13T15:47:59.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14090 9.88 MB/sec warmup 95 sec latency 337.015 ms 2024-09-13T15:48:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:47:59 smithi090 ceph-mon[27811]: pgmap v680: 129 pgs: 129 active+clean; 410 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 190 op/s 2024-09-13T15:48:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:47:59 smithi175 ceph-mon[36128]: pgmap v680: 129 pgs: 129 active+clean; 410 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 190 op/s 2024-09-13T15:48:00.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 25556 16.15 MB/sec warmup 96 sec latency 616.084 ms 2024-09-13T15:48:00.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14243 9.95 MB/sec warmup 96 sec latency 175.989 ms 2024-09-13T15:48:01.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 25565 15.99 MB/sec warmup 97 sec latency 621.671 ms 2024-09-13T15:48:01.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14400 9.95 MB/sec warmup 97 sec latency 124.355 ms 2024-09-13T15:48:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:01 smithi090 ceph-mon[27811]: pgmap v681: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.5 MiB/s wr, 126 op/s 2024-09-13T15:48:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:01 smithi175 ceph-mon[36128]: pgmap v681: 129 pgs: 129 active+clean; 412 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.5 MiB/s wr, 126 op/s 2024-09-13T15:48:02.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 25594 15.83 MB/sec warmup 98 sec latency 582.144 ms 2024-09-13T15:48:02.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14510 9.92 MB/sec warmup 98 sec latency 176.641 ms 2024-09-13T15:48:03.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 25612 15.68 MB/sec warmup 99 sec latency 486.019 ms 2024-09-13T15:48:03.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14618 9.84 MB/sec warmup 99 sec latency 109.330 ms 2024-09-13T15:48:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:03 smithi090 ceph-mon[27811]: pgmap v682: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.2 MiB/s wr, 131 op/s 2024-09-13T15:48:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:03 smithi175 ceph-mon[36128]: pgmap v682: 129 pgs: 129 active+clean; 414 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.2 MiB/s wr, 131 op/s 2024-09-13T15:48:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:04.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 25741 15.55 MB/sec warmup 100 sec latency 424.864 ms 2024-09-13T15:48:04.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14722 9.75 MB/sec warmup 100 sec latency 105.197 ms 2024-09-13T15:48:05.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 26666 15.86 MB/sec warmup 101 sec latency 154.978 ms 2024-09-13T15:48:05.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 14849 9.67 MB/sec warmup 101 sec latency 117.378 ms 2024-09-13T15:48:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:05 smithi090 ceph-mon[27811]: pgmap v683: 129 pgs: 129 active+clean; 413 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 148 op/s 2024-09-13T15:48:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:05 smithi175 ceph-mon[36128]: pgmap v683: 129 pgs: 129 active+clean; 413 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 148 op/s 2024-09-13T15:48:06.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 27784 16.37 MB/sec warmup 102 sec latency 167.791 ms 2024-09-13T15:48:06.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15013 9.62 MB/sec warmup 102 sec latency 83.848 ms 2024-09-13T15:48:07.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29098 16.91 MB/sec warmup 103 sec latency 125.622 ms 2024-09-13T15:48:07.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15159 9.57 MB/sec warmup 103 sec latency 126.649 ms 2024-09-13T15:48:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:07 smithi090 ceph-mon[27811]: pgmap v684: 129 pgs: 129 active+clean; 410 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.3 MiB/s wr, 118 op/s 2024-09-13T15:48:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:07 smithi175 ceph-mon[36128]: pgmap v684: 129 pgs: 129 active+clean; 410 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.3 MiB/s wr, 118 op/s 2024-09-13T15:48:08.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29154 16.76 MB/sec warmup 104 sec latency 423.504 ms 2024-09-13T15:48:08.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15314 9.59 MB/sec warmup 104 sec latency 174.224 ms 2024-09-13T15:48:09.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29164 16.60 MB/sec warmup 105 sec latency 543.563 ms 2024-09-13T15:48:09.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15453 9.56 MB/sec warmup 105 sec latency 142.920 ms 2024-09-13T15:48:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:09 smithi090 ceph-mon[27811]: pgmap v685: 129 pgs: 129 active+clean; 397 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 4.3 MiB/s wr, 182 op/s 2024-09-13T15:48:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:09 smithi175 ceph-mon[36128]: pgmap v685: 129 pgs: 129 active+clean; 397 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 4.3 MiB/s wr, 182 op/s 2024-09-13T15:48:10.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29195 16.45 MB/sec warmup 106 sec latency 538.538 ms 2024-09-13T15:48:10.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15579 9.51 MB/sec warmup 106 sec latency 192.000 ms 2024-09-13T15:48:11.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29220 16.31 MB/sec warmup 107 sec latency 604.585 ms 2024-09-13T15:48:11.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15729 9.59 MB/sec warmup 107 sec latency 184.837 ms 2024-09-13T15:48:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:11 smithi090 ceph-mon[27811]: pgmap v686: 129 pgs: 129 active+clean; 394 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.3 MiB/s wr, 150 op/s 2024-09-13T15:48:12.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:11 smithi175 ceph-mon[36128]: pgmap v686: 129 pgs: 129 active+clean; 394 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.3 MiB/s wr, 150 op/s 2024-09-13T15:48:12.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29224 16.16 MB/sec warmup 108 sec latency 607.967 ms 2024-09-13T15:48:12.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15856 9.51 MB/sec warmup 108 sec latency 222.435 ms 2024-09-13T15:48:13.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29270 16.02 MB/sec warmup 109 sec latency 598.739 ms 2024-09-13T15:48:13.394 INFO:tasks.workunit.client.1.smithi175.stdout: 10 15998 9.43 MB/sec warmup 109 sec latency 146.816 ms 2024-09-13T15:48:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:13 smithi090 ceph-mon[27811]: pgmap v687: 129 pgs: 129 active+clean; 392 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.8 MiB/s wr, 209 op/s 2024-09-13T15:48:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:13 smithi175 ceph-mon[36128]: pgmap v687: 129 pgs: 129 active+clean; 392 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.8 MiB/s wr, 209 op/s 2024-09-13T15:48:14.358 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29310 15.88 MB/sec warmup 110 sec latency 623.199 ms 2024-09-13T15:48:14.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 16171 9.37 MB/sec warmup 110 sec latency 117.569 ms 2024-09-13T15:48:15.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29339 15.74 MB/sec warmup 111 sec latency 717.138 ms 2024-09-13T15:48:15.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 16305 9.31 MB/sec warmup 111 sec latency 241.837 ms 2024-09-13T15:48:16.188 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:15 smithi090 ceph-mon[27811]: pgmap v688: 129 pgs: 129 active+clean; 392 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.2 MiB/s wr, 212 op/s 2024-09-13T15:48:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:15 smithi175 ceph-mon[36128]: pgmap v688: 129 pgs: 129 active+clean; 392 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.2 MiB/s wr, 212 op/s 2024-09-13T15:48:16.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29438 15.65 MB/sec warmup 112 sec latency 558.401 ms 2024-09-13T15:48:16.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 16452 9.26 MB/sec warmup 112 sec latency 117.600 ms 2024-09-13T15:48:17.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 29845 15.74 MB/sec warmup 113 sec latency 608.783 ms 2024-09-13T15:48:17.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 16607 9.26 MB/sec warmup 113 sec latency 123.606 ms 2024-09-13T15:48:18.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:17 smithi175 ceph-mon[36128]: pgmap v689: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 203 op/s 2024-09-13T15:48:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:17 smithi090 ceph-mon[27811]: pgmap v689: 129 pgs: 129 active+clean; 393 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.5 MiB/s wr, 203 op/s 2024-09-13T15:48:18.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 30036 15.72 MB/sec warmup 114 sec latency 708.550 ms 2024-09-13T15:48:18.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 16746 9.27 MB/sec warmup 114 sec latency 162.050 ms 2024-09-13T15:48:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:19.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 30233 15.59 MB/sec warmup 115 sec latency 835.808 ms 2024-09-13T15:48:19.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 16927 9.32 MB/sec warmup 115 sec latency 134.668 ms 2024-09-13T15:48:20.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:19 smithi175 ceph-mon[36128]: pgmap v690: 129 pgs: 129 active+clean; 392 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.8 MiB/s wr, 215 op/s 2024-09-13T15:48:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:19 smithi090 ceph-mon[27811]: pgmap v690: 129 pgs: 129 active+clean; 392 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.5 KiB/s rd, 2.8 MiB/s wr, 215 op/s 2024-09-13T15:48:20.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 30878 15.67 MB/sec warmup 116 sec latency 703.840 ms 2024-09-13T15:48:20.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 17129 9.44 MB/sec warmup 116 sec latency 160.865 ms 2024-09-13T15:48:21.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 31373 15.92 MB/sec warmup 117 sec latency 478.644 ms 2024-09-13T15:48:21.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 17369 9.48 MB/sec warmup 117 sec latency 89.261 ms 2024-09-13T15:48:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:21 smithi090 ceph-mon[27811]: pgmap v691: 129 pgs: 129 active+clean; 394 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-09-13T15:48:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:21 smithi175 ceph-mon[36128]: pgmap v691: 129 pgs: 129 active+clean; 394 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.4 MiB/s wr, 167 op/s 2024-09-13T15:48:22.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 32283 16.21 MB/sec warmup 118 sec latency 423.018 ms 2024-09-13T15:48:22.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 17504 9.48 MB/sec warmup 118 sec latency 114.815 ms 2024-09-13T15:48:23.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 32712 16.26 MB/sec warmup 119 sec latency 111.560 ms 2024-09-13T15:48:23.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 17591 9.46 MB/sec warmup 119 sec latency 86.946 ms 2024-09-13T15:48:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:23 smithi090 ceph-mon[27811]: pgmap v692: 129 pgs: 129 active+clean; 407 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-09-13T15:48:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:23 smithi175 ceph-mon[36128]: pgmap v692: 129 pgs: 129 active+clean; 407 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.6 MiB/s wr, 177 op/s 2024-09-13T15:48:25.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 34132 55.42 MB/sec execute 1 sec latency 191.658 ms 2024-09-13T15:48:25.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 17916 21.51 MB/sec execute 1 sec latency 87.340 ms 2024-09-13T15:48:26.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:25 smithi090 ceph-mon[27811]: pgmap v693: 129 pgs: 129 active+clean; 415 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 147 op/s 2024-09-13T15:48:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:25 smithi175 ceph-mon[36128]: pgmap v693: 129 pgs: 129 active+clean; 415 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 147 op/s 2024-09-13T15:48:26.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 34362 31.66 MB/sec execute 2 sec latency 524.595 ms 2024-09-13T15:48:26.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18045 14.58 MB/sec execute 2 sec latency 148.938 ms 2024-09-13T15:48:27.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 34604 29.51 MB/sec execute 3 sec latency 715.436 ms 2024-09-13T15:48:27.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18122 9.74 MB/sec execute 3 sec latency 163.332 ms 2024-09-13T15:48:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:27 smithi090 ceph-mon[27811]: pgmap v694: 129 pgs: 129 active+clean; 416 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.1 MiB/s wr, 140 op/s 2024-09-13T15:48:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:27 smithi175 ceph-mon[36128]: pgmap v694: 129 pgs: 129 active+clean; 416 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.1 MiB/s wr, 140 op/s 2024-09-13T15:48:28.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 34818 26.62 MB/sec execute 4 sec latency 685.333 ms 2024-09-13T15:48:28.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18212 7.87 MB/sec execute 4 sec latency 141.977 ms 2024-09-13T15:48:29.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 35082 24.87 MB/sec execute 5 sec latency 666.144 ms 2024-09-13T15:48:29.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18302 6.42 MB/sec execute 5 sec latency 184.931 ms 2024-09-13T15:48:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:29 smithi090 ceph-mon[27811]: pgmap v695: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 154 op/s 2024-09-13T15:48:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:29 smithi175 ceph-mon[36128]: pgmap v695: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.9 MiB/s wr, 154 op/s 2024-09-13T15:48:30.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 35182 21.78 MB/sec execute 6 sec latency 535.160 ms 2024-09-13T15:48:30.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18390 5.54 MB/sec execute 6 sec latency 193.355 ms 2024-09-13T15:48:31.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 35272 19.15 MB/sec execute 7 sec latency 633.957 ms 2024-09-13T15:48:31.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18481 5.01 MB/sec execute 7 sec latency 248.286 ms 2024-09-13T15:48:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:31 smithi090 ceph-mon[27811]: pgmap v696: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.7 MiB/s wr, 136 op/s 2024-09-13T15:48:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:31 smithi175 ceph-mon[36128]: pgmap v696: 129 pgs: 129 active+clean; 422 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.7 MiB/s wr, 136 op/s 2024-09-13T15:48:32.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 35825 20.71 MB/sec execute 8 sec latency 779.013 ms 2024-09-13T15:48:32.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18640 4.96 MB/sec execute 8 sec latency 129.365 ms 2024-09-13T15:48:33.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 36011 19.50 MB/sec execute 9 sec latency 712.039 ms 2024-09-13T15:48:33.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18770 5.20 MB/sec execute 9 sec latency 171.120 ms 2024-09-13T15:48:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:33 smithi090 ceph-mon[27811]: pgmap v697: 129 pgs: 129 active+clean; 407 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.5 MiB/s wr, 132 op/s 2024-09-13T15:48:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:33 smithi175 ceph-mon[36128]: pgmap v697: 129 pgs: 129 active+clean; 407 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.5 MiB/s wr, 132 op/s 2024-09-13T15:48:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:34.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 36191 18.54 MB/sec execute 10 sec latency 704.463 ms 2024-09-13T15:48:34.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 18906 5.70 MB/sec execute 10 sec latency 223.819 ms 2024-09-13T15:48:35.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 36220 16.92 MB/sec execute 11 sec latency 544.629 ms 2024-09-13T15:48:35.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 19042 5.81 MB/sec execute 11 sec latency 145.042 ms 2024-09-13T15:48:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:35 smithi090 ceph-mon[27811]: pgmap v698: 129 pgs: 129 active+clean; 404 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.7 MiB/s wr, 175 op/s 2024-09-13T15:48:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:35 smithi175 ceph-mon[36128]: pgmap v698: 129 pgs: 129 active+clean; 404 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.7 MiB/s wr, 175 op/s 2024-09-13T15:48:36.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 36229 15.53 MB/sec execute 12 sec latency 521.952 ms 2024-09-13T15:48:36.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 19167 5.42 MB/sec execute 12 sec latency 150.774 ms 2024-09-13T15:48:37.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 36246 14.36 MB/sec execute 13 sec latency 614.413 ms 2024-09-13T15:48:37.395 INFO:tasks.workunit.client.1.smithi175.stdout: 10 19330 6.37 MB/sec execute 13 sec latency 224.486 ms 2024-09-13T15:48:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:37 smithi090 ceph-mon[27811]: pgmap v699: 129 pgs: 129 active+clean; 402 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.4 MiB/s wr, 177 op/s 2024-09-13T15:48:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:37 smithi175 ceph-mon[36128]: pgmap v699: 129 pgs: 129 active+clean; 402 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.4 MiB/s wr, 177 op/s 2024-09-13T15:48:38.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 36361 13.54 MB/sec execute 14 sec latency 499.416 ms 2024-09-13T15:48:38.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 19500 6.03 MB/sec execute 14 sec latency 77.022 ms 2024-09-13T15:48:39.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 37267 15.76 MB/sec execute 15 sec latency 146.663 ms 2024-09-13T15:48:39.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 19710 5.82 MB/sec execute 15 sec latency 87.562 ms 2024-09-13T15:48:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:39 smithi090 ceph-mon[27811]: pgmap v700: 129 pgs: 129 active+clean; 395 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.1 MiB/s wr, 195 op/s 2024-09-13T15:48:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:39 smithi175 ceph-mon[36128]: pgmap v700: 129 pgs: 129 active+clean; 395 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.1 MiB/s wr, 195 op/s 2024-09-13T15:48:40.359 INFO:tasks.workunit.client.0.smithi090.stdout: 10 38362 18.61 MB/sec execute 16 sec latency 126.239 ms 2024-09-13T15:48:40.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 19908 5.73 MB/sec execute 16 sec latency 60.319 ms 2024-09-13T15:48:41.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39465 21.28 MB/sec execute 17 sec latency 265.982 ms 2024-09-13T15:48:41.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 20081 5.66 MB/sec execute 17 sec latency 74.081 ms 2024-09-13T15:48:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:41 smithi090 ceph-mon[27811]: pgmap v701: 129 pgs: 129 active+clean; 395 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.3 MiB/s wr, 152 op/s 2024-09-13T15:48:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:41 smithi175 ceph-mon[36128]: pgmap v701: 129 pgs: 129 active+clean; 395 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.3 MiB/s wr, 152 op/s 2024-09-13T15:48:42.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39794 21.00 MB/sec execute 18 sec latency 167.410 ms 2024-09-13T15:48:42.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 20283 6.26 MB/sec execute 18 sec latency 100.183 ms 2024-09-13T15:48:43.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39842 19.95 MB/sec execute 19 sec latency 556.046 ms 2024-09-13T15:48:43.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 20429 6.46 MB/sec execute 19 sec latency 135.420 ms 2024-09-13T15:48:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: pgmap v702: 129 pgs: 129 active+clean; 401 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.7 MiB/s wr, 212 op/s 2024-09-13T15:48:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:48:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:48:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:48:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:48:44.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:48:44.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:43 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:48:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: pgmap v702: 129 pgs: 129 active+clean; 401 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.7 MiB/s wr, 212 op/s 2024-09-13T15:48:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:48:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:48:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:48:44.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:48:44.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:48:44.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:43 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:48:44.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39862 18.98 MB/sec execute 20 sec latency 601.625 ms 2024-09-13T15:48:44.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 20615 6.85 MB/sec execute 20 sec latency 171.474 ms 2024-09-13T15:48:45.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39883 18.09 MB/sec execute 21 sec latency 623.783 ms 2024-09-13T15:48:45.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 20846 7.70 MB/sec execute 21 sec latency 172.135 ms 2024-09-13T15:48:46.204 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:45 smithi090 ceph-mon[27811]: pgmap v703: 129 pgs: 129 active+clean; 400 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.1 MiB/s wr, 215 op/s 2024-09-13T15:48:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:45 smithi175 ceph-mon[36128]: pgmap v703: 129 pgs: 129 active+clean; 400 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.1 MiB/s wr, 215 op/s 2024-09-13T15:48:46.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39926 17.32 MB/sec execute 22 sec latency 566.867 ms 2024-09-13T15:48:46.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21023 7.72 MB/sec execute 22 sec latency 150.879 ms 2024-09-13T15:48:47.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 39976 16.62 MB/sec execute 23 sec latency 560.577 ms 2024-09-13T15:48:47.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21110 7.77 MB/sec execute 23 sec latency 152.223 ms 2024-09-13T15:48:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:47 smithi090 ceph-mon[27811]: pgmap v704: 129 pgs: 129 active+clean; 408 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.5 MiB/s wr, 166 op/s 2024-09-13T15:48:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:47 smithi175 ceph-mon[36128]: pgmap v704: 129 pgs: 129 active+clean; 408 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.5 MiB/s wr, 166 op/s 2024-09-13T15:48:48.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 40013 15.97 MB/sec execute 24 sec latency 561.980 ms 2024-09-13T15:48:48.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21177 7.64 MB/sec execute 24 sec latency 139.716 ms 2024-09-13T15:48:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:48:49.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 40484 16.55 MB/sec execute 25 sec latency 535.813 ms 2024-09-13T15:48:49.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21270 7.65 MB/sec execute 25 sec latency 126.096 ms 2024-09-13T15:48:50.180 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:49 smithi175 ceph-mon[36128]: pgmap v705: 129 pgs: 129 active+clean; 416 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 169 op/s 2024-09-13T15:48:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:49 smithi090 ceph-mon[27811]: pgmap v705: 129 pgs: 129 active+clean; 416 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.8 MiB/s wr, 169 op/s 2024-09-13T15:48:50.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 40800 16.44 MB/sec execute 26 sec latency 558.636 ms 2024-09-13T15:48:50.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21424 8.14 MB/sec execute 26 sec latency 132.555 ms 2024-09-13T15:48:51.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 40872 15.85 MB/sec execute 27 sec latency 821.107 ms 2024-09-13T15:48:51.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21551 7.94 MB/sec execute 27 sec latency 174.687 ms 2024-09-13T15:48:52.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:51 smithi175 ceph-mon[36128]: pgmap v706: 129 pgs: 129 active+clean; 418 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-09-13T15:48:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:51 smithi090 ceph-mon[27811]: pgmap v706: 129 pgs: 129 active+clean; 418 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.7 MiB/s wr, 146 op/s 2024-09-13T15:48:52.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 41221 15.63 MB/sec execute 28 sec latency 768.260 ms 2024-09-13T15:48:52.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21664 7.89 MB/sec execute 28 sec latency 220.240 ms 2024-09-13T15:48:53.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 41298 15.17 MB/sec execute 29 sec latency 661.229 ms 2024-09-13T15:48:53.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21768 7.68 MB/sec execute 29 sec latency 102.916 ms 2024-09-13T15:48:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:53 smithi090 ceph-mon[27811]: pgmap v707: 129 pgs: 129 active+clean; 420 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.2 MiB/s wr, 151 op/s 2024-09-13T15:48:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:53 smithi175 ceph-mon[36128]: pgmap v707: 129 pgs: 129 active+clean; 420 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.6 KiB/s rd, 4.2 MiB/s wr, 151 op/s 2024-09-13T15:48:54.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 41547 15.18 MB/sec execute 30 sec latency 684.772 ms 2024-09-13T15:48:54.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 21881 7.46 MB/sec execute 30 sec latency 105.821 ms 2024-09-13T15:48:55.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 42645 17.20 MB/sec execute 31 sec latency 487.029 ms 2024-09-13T15:48:55.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22004 7.30 MB/sec execute 31 sec latency 140.378 ms 2024-09-13T15:48:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:55 smithi090 ceph-mon[27811]: pgmap v708: 129 pgs: 129 active+clean; 424 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.5 MiB/s wr, 133 op/s 2024-09-13T15:48:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:55 smithi175 ceph-mon[36128]: pgmap v708: 129 pgs: 129 active+clean; 424 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.5 MiB/s wr, 133 op/s 2024-09-13T15:48:56.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 43335 17.78 MB/sec execute 32 sec latency 139.474 ms 2024-09-13T15:48:56.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22149 7.18 MB/sec execute 32 sec latency 120.967 ms 2024-09-13T15:48:57.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 43451 17.33 MB/sec execute 33 sec latency 254.492 ms 2024-09-13T15:48:57.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22328 7.21 MB/sec execute 33 sec latency 136.664 ms 2024-09-13T15:48:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:57 smithi090 ceph-mon[27811]: pgmap v709: 129 pgs: 129 active+clean; 422 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.2 MiB/s wr, 116 op/s 2024-09-13T15:48:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:57 smithi175 ceph-mon[36128]: pgmap v709: 129 pgs: 129 active+clean; 422 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.2 MiB/s wr, 116 op/s 2024-09-13T15:48:58.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 44092 17.81 MB/sec execute 34 sec latency 275.895 ms 2024-09-13T15:48:58.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22478 7.23 MB/sec execute 34 sec latency 117.621 ms 2024-09-13T15:48:59.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 45087 18.48 MB/sec execute 35 sec latency 334.156 ms 2024-09-13T15:48:59.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22660 7.31 MB/sec execute 35 sec latency 70.893 ms 2024-09-13T15:49:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:48:59 smithi090 ceph-mon[27811]: pgmap v710: 129 pgs: 129 active+clean; 421 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.0 MiB/s wr, 183 op/s 2024-09-13T15:49:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:48:59 smithi175 ceph-mon[36128]: pgmap v710: 129 pgs: 129 active+clean; 421 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 4.0 MiB/s wr, 183 op/s 2024-09-13T15:49:00.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 45268 18.48 MB/sec execute 36 sec latency 757.391 ms 2024-09-13T15:49:00.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22800 7.38 MB/sec execute 36 sec latency 316.458 ms 2024-09-13T15:49:01.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 45454 18.38 MB/sec execute 37 sec latency 702.042 ms 2024-09-13T15:49:01.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 22971 7.43 MB/sec execute 37 sec latency 119.223 ms 2024-09-13T15:49:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:01 smithi090 ceph-mon[27811]: pgmap v711: 129 pgs: 129 active+clean; 415 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-09-13T15:49:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:01 smithi175 ceph-mon[36128]: pgmap v711: 129 pgs: 129 active+clean; 415 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.4 MiB/s wr, 151 op/s 2024-09-13T15:49:02.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 45689 18.35 MB/sec execute 38 sec latency 749.158 ms 2024-09-13T15:49:02.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23090 7.27 MB/sec execute 38 sec latency 165.423 ms 2024-09-13T15:49:03.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 45785 18.01 MB/sec execute 39 sec latency 587.891 ms 2024-09-13T15:49:03.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23218 7.14 MB/sec execute 39 sec latency 259.196 ms 2024-09-13T15:49:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:03 smithi175 ceph-mon[36128]: pgmap v712: 129 pgs: 129 active+clean; 401 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.6 MiB/s wr, 186 op/s 2024-09-13T15:49:04.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:03 smithi090 ceph-mon[27811]: pgmap v712: 129 pgs: 129 active+clean; 401 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.6 MiB/s wr, 186 op/s 2024-09-13T15:49:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:04.360 INFO:tasks.workunit.client.0.smithi090.stdout: 10 45884 17.66 MB/sec execute 40 sec latency 673.703 ms 2024-09-13T15:49:04.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23365 7.02 MB/sec execute 40 sec latency 237.658 ms 2024-09-13T15:49:05.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46457 18.03 MB/sec execute 41 sec latency 772.750 ms 2024-09-13T15:49:05.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23530 6.96 MB/sec execute 41 sec latency 126.869 ms 2024-09-13T15:49:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:05 smithi090 ceph-mon[27811]: pgmap v713: 129 pgs: 129 active+clean; 399 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.4 MiB/s wr, 233 op/s 2024-09-13T15:49:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:05 smithi175 ceph-mon[36128]: pgmap v713: 129 pgs: 129 active+clean; 399 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.4 MiB/s wr, 233 op/s 2024-09-13T15:49:06.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46560 17.72 MB/sec execute 42 sec latency 696.474 ms 2024-09-13T15:49:06.396 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23644 6.87 MB/sec execute 42 sec latency 177.577 ms 2024-09-13T15:49:07.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46831 17.65 MB/sec execute 43 sec latency 542.996 ms 2024-09-13T15:49:07.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23821 7.06 MB/sec execute 43 sec latency 165.220 ms 2024-09-13T15:49:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:07 smithi090 ceph-mon[27811]: pgmap v714: 129 pgs: 129 active+clean; 400 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.6 MiB/s wr, 203 op/s 2024-09-13T15:49:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:07 smithi175 ceph-mon[36128]: pgmap v714: 129 pgs: 129 active+clean; 400 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.6 MiB/s wr, 203 op/s 2024-09-13T15:49:08.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46856 17.26 MB/sec execute 44 sec latency 557.326 ms 2024-09-13T15:49:08.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 23963 7.10 MB/sec execute 44 sec latency 146.082 ms 2024-09-13T15:49:09.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46869 16.89 MB/sec execute 45 sec latency 541.897 ms 2024-09-13T15:49:09.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 24150 7.29 MB/sec execute 45 sec latency 141.142 ms 2024-09-13T15:49:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:09 smithi090 ceph-mon[27811]: pgmap v715: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-09-13T15:49:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:09 smithi175 ceph-mon[36128]: pgmap v715: 129 pgs: 129 active+clean; 403 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 8.6 KiB/s rd, 2.8 MiB/s wr, 212 op/s 2024-09-13T15:49:10.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46886 16.53 MB/sec execute 46 sec latency 534.894 ms 2024-09-13T15:49:10.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 24346 7.58 MB/sec execute 46 sec latency 133.332 ms 2024-09-13T15:49:11.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46918 16.19 MB/sec execute 47 sec latency 545.644 ms 2024-09-13T15:49:11.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 24479 7.61 MB/sec execute 47 sec latency 227.373 ms 2024-09-13T15:49:11.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:11 smithi090 ceph-mon[27811]: pgmap v716: 129 pgs: 129 active+clean; 406 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-09-13T15:49:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:11 smithi175 ceph-mon[36128]: pgmap v716: 129 pgs: 129 active+clean; 406 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.5 MiB/s wr, 161 op/s 2024-09-13T15:49:12.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 46938 15.86 MB/sec execute 48 sec latency 509.473 ms 2024-09-13T15:49:12.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 24648 7.71 MB/sec execute 48 sec latency 122.817 ms 2024-09-13T15:49:13.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 47260 15.88 MB/sec execute 49 sec latency 193.715 ms 2024-09-13T15:49:13.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 24737 7.71 MB/sec execute 49 sec latency 99.872 ms 2024-09-13T15:49:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:13 smithi090 ceph-mon[27811]: pgmap v717: 129 pgs: 129 active+clean; 420 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.6 MiB/s wr, 171 op/s 2024-09-13T15:49:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:13 smithi175 ceph-mon[36128]: pgmap v717: 129 pgs: 129 active+clean; 420 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 3.6 MiB/s wr, 171 op/s 2024-09-13T15:49:14.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 48414 16.49 MB/sec execute 50 sec latency 116.977 ms 2024-09-13T15:49:14.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 24832 7.66 MB/sec execute 50 sec latency 96.802 ms 2024-09-13T15:49:15.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 49464 17.67 MB/sec execute 51 sec latency 120.894 ms 2024-09-13T15:49:15.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25018 7.91 MB/sec execute 51 sec latency 79.606 ms 2024-09-13T15:49:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:15 smithi090 ceph-mon[27811]: pgmap v718: 129 pgs: 129 active+clean; 421 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-13T15:49:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:15 smithi175 ceph-mon[36128]: pgmap v718: 129 pgs: 129 active+clean; 421 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-13T15:49:16.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 50424 18.30 MB/sec execute 52 sec latency 200.299 ms 2024-09-13T15:49:16.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25138 7.83 MB/sec execute 52 sec latency 271.320 ms 2024-09-13T15:49:17.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 50543 18.01 MB/sec execute 53 sec latency 125.725 ms 2024-09-13T15:49:17.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25267 7.79 MB/sec execute 53 sec latency 98.352 ms 2024-09-13T15:49:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:17 smithi090 ceph-mon[27811]: pgmap v719: 129 pgs: 129 active+clean; 423 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.9 MiB/s wr, 115 op/s 2024-09-13T15:49:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:17 smithi175 ceph-mon[36128]: pgmap v719: 129 pgs: 129 active+clean; 423 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.9 MiB/s wr, 115 op/s 2024-09-13T15:49:18.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 50609 17.70 MB/sec execute 54 sec latency 520.643 ms 2024-09-13T15:49:18.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25371 7.68 MB/sec execute 54 sec latency 167.288 ms 2024-09-13T15:49:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:19.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 50655 17.40 MB/sec execute 55 sec latency 716.697 ms 2024-09-13T15:49:19.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25463 7.56 MB/sec execute 55 sec latency 157.363 ms 2024-09-13T15:49:20.167 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:19 smithi175 ceph-mon[36128]: pgmap v720: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.6 MiB/s wr, 155 op/s 2024-09-13T15:49:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:19 smithi090 ceph-mon[27811]: pgmap v720: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.6 MiB/s wr, 155 op/s 2024-09-13T15:49:20.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 51089 17.62 MB/sec execute 56 sec latency 560.696 ms 2024-09-13T15:49:20.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25572 7.47 MB/sec execute 56 sec latency 156.130 ms 2024-09-13T15:49:21.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 51384 17.55 MB/sec execute 57 sec latency 786.635 ms 2024-09-13T15:49:21.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25711 7.39 MB/sec execute 57 sec latency 119.287 ms 2024-09-13T15:49:22.117 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:21 smithi090 ceph-mon[27811]: pgmap v721: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.4 MiB/s wr, 137 op/s 2024-09-13T15:49:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:21 smithi175 ceph-mon[36128]: pgmap v721: 129 pgs: 129 active+clean; 428 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.4 MiB/s wr, 137 op/s 2024-09-13T15:49:22.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 51507 17.27 MB/sec execute 58 sec latency 797.658 ms 2024-09-13T15:49:22.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25837 7.38 MB/sec execute 58 sec latency 149.459 ms 2024-09-13T15:49:23.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 51777 17.09 MB/sec execute 59 sec latency 739.794 ms 2024-09-13T15:49:23.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 25985 7.36 MB/sec execute 59 sec latency 162.258 ms 2024-09-13T15:49:24.142 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:23 smithi175 ceph-mon[36128]: pgmap v722: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.5 MiB/s wr, 139 op/s 2024-09-13T15:49:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:23 smithi090 ceph-mon[27811]: pgmap v722: 129 pgs: 129 active+clean; 413 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.5 MiB/s wr, 139 op/s 2024-09-13T15:49:24.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 51931 16.89 MB/sec execute 60 sec latency 638.886 ms 2024-09-13T15:49:24.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 26132 7.44 MB/sec execute 60 sec latency 142.285 ms 2024-09-13T15:49:25.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 52232 16.96 MB/sec execute 61 sec latency 722.696 ms 2024-09-13T15:49:25.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 26276 7.38 MB/sec execute 61 sec latency 284.727 ms 2024-09-13T15:49:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:25 smithi090 ceph-mon[27811]: pgmap v723: 129 pgs: 129 active+clean; 410 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.7 MiB/s wr, 188 op/s 2024-09-13T15:49:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:25 smithi175 ceph-mon[36128]: pgmap v723: 129 pgs: 129 active+clean; 410 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.7 MiB/s wr, 188 op/s 2024-09-13T15:49:26.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 52412 16.98 MB/sec execute 62 sec latency 784.708 ms 2024-09-13T15:49:26.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 26459 7.52 MB/sec execute 62 sec latency 181.301 ms 2024-09-13T15:49:27.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 52612 16.97 MB/sec execute 63 sec latency 824.684 ms 2024-09-13T15:49:27.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 26611 7.42 MB/sec execute 63 sec latency 123.147 ms 2024-09-13T15:49:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:27 smithi090 ceph-mon[27811]: pgmap v724: 129 pgs: 129 active+clean; 404 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-09-13T15:49:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:27 smithi175 ceph-mon[36128]: pgmap v724: 129 pgs: 129 active+clean; 404 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.4 MiB/s wr, 178 op/s 2024-09-13T15:49:28.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 52794 16.89 MB/sec execute 64 sec latency 513.135 ms 2024-09-13T15:49:28.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 26737 7.34 MB/sec execute 64 sec latency 279.129 ms 2024-09-13T15:49:29.361 INFO:tasks.workunit.client.0.smithi090.stdout: 10 52947 16.74 MB/sec execute 65 sec latency 709.976 ms 2024-09-13T15:49:29.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 26882 7.26 MB/sec execute 65 sec latency 239.168 ms 2024-09-13T15:49:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:29 smithi090 ceph-mon[27811]: pgmap v725: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.4 MiB/s wr, 229 op/s 2024-09-13T15:49:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:29 smithi175 ceph-mon[36128]: pgmap v725: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.4 MiB/s wr, 229 op/s 2024-09-13T15:49:30.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 53366 16.84 MB/sec execute 66 sec latency 536.750 ms 2024-09-13T15:49:30.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 27013 7.20 MB/sec execute 66 sec latency 153.479 ms 2024-09-13T15:49:31.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 53935 17.04 MB/sec execute 67 sec latency 622.891 ms 2024-09-13T15:49:31.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 27195 7.21 MB/sec execute 67 sec latency 73.631 ms 2024-09-13T15:49:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:31 smithi090 ceph-mon[27811]: pgmap v726: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-09-13T15:49:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:31 smithi175 ceph-mon[36128]: pgmap v726: 129 pgs: 129 active+clean; 402 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 2.6 MiB/s wr, 178 op/s 2024-09-13T15:49:32.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 54042 16.83 MB/sec execute 68 sec latency 200.637 ms 2024-09-13T15:49:32.397 INFO:tasks.workunit.client.1.smithi175.stdout: 10 27409 7.30 MB/sec execute 68 sec latency 104.263 ms 2024-09-13T15:49:33.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 54635 17.07 MB/sec execute 69 sec latency 118.177 ms 2024-09-13T15:49:33.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 27604 7.42 MB/sec execute 69 sec latency 84.007 ms 2024-09-13T15:49:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:33 smithi090 ceph-mon[27811]: pgmap v727: 129 pgs: 129 active+clean; 408 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-09-13T15:49:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:33 smithi175 ceph-mon[36128]: pgmap v727: 129 pgs: 129 active+clean; 408 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.0 MiB/s wr, 205 op/s 2024-09-13T15:49:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:34.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 55750 17.46 MB/sec execute 70 sec latency 196.663 ms 2024-09-13T15:49:34.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 27779 7.48 MB/sec execute 70 sec latency 83.962 ms 2024-09-13T15:49:35.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 56824 18.28 MB/sec execute 71 sec latency 133.359 ms 2024-09-13T15:49:35.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28010 7.73 MB/sec execute 71 sec latency 70.337 ms 2024-09-13T15:49:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:35 smithi090 ceph-mon[27811]: pgmap v728: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-09-13T15:49:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:35 smithi175 ceph-mon[36128]: pgmap v728: 129 pgs: 129 active+clean; 412 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-09-13T15:49:36.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57465 18.51 MB/sec execute 72 sec latency 461.268 ms 2024-09-13T15:49:36.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28207 7.80 MB/sec execute 72 sec latency 105.571 ms 2024-09-13T15:49:37.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57499 18.27 MB/sec execute 73 sec latency 610.906 ms 2024-09-13T15:49:37.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28292 7.79 MB/sec execute 73 sec latency 114.858 ms 2024-09-13T15:49:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:37 smithi090 ceph-mon[27811]: pgmap v729: 129 pgs: 129 active+clean; 416 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-09-13T15:49:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:37 smithi175 ceph-mon[36128]: pgmap v729: 129 pgs: 129 active+clean; 416 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.2 MiB/s wr, 160 op/s 2024-09-13T15:49:38.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57512 18.02 MB/sec execute 74 sec latency 534.895 ms 2024-09-13T15:49:38.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28368 7.75 MB/sec execute 74 sec latency 142.668 ms 2024-09-13T15:49:39.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57521 17.79 MB/sec execute 75 sec latency 528.144 ms 2024-09-13T15:49:39.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28462 7.73 MB/sec execute 75 sec latency 112.848 ms 2024-09-13T15:49:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:39 smithi090 ceph-mon[27811]: pgmap v730: 129 pgs: 129 active+clean; 424 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 185 op/s 2024-09-13T15:49:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:39 smithi175 ceph-mon[36128]: pgmap v730: 129 pgs: 129 active+clean; 424 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.1 MiB/s wr, 185 op/s 2024-09-13T15:49:40.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57546 17.56 MB/sec execute 76 sec latency 557.834 ms 2024-09-13T15:49:40.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28599 7.85 MB/sec execute 76 sec latency 233.033 ms 2024-09-13T15:49:41.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57578 17.34 MB/sec execute 77 sec latency 647.967 ms 2024-09-13T15:49:41.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28733 7.80 MB/sec execute 77 sec latency 142.905 ms 2024-09-13T15:49:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:41 smithi090 ceph-mon[27811]: pgmap v731: 129 pgs: 129 active+clean; 425 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.8 MiB/s wr, 128 op/s 2024-09-13T15:49:42.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:41 smithi175 ceph-mon[36128]: pgmap v731: 129 pgs: 129 active+clean; 425 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.8 MiB/s wr, 128 op/s 2024-09-13T15:49:42.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57588 17.12 MB/sec execute 78 sec latency 563.569 ms 2024-09-13T15:49:42.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28836 7.75 MB/sec execute 78 sec latency 147.859 ms 2024-09-13T15:49:43.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57619 16.92 MB/sec execute 79 sec latency 539.011 ms 2024-09-13T15:49:43.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 28942 7.68 MB/sec execute 79 sec latency 151.727 ms 2024-09-13T15:49:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:43 smithi090 ceph-mon[27811]: pgmap v732: 129 pgs: 129 active+clean; 426 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.4 MiB/s wr, 136 op/s 2024-09-13T15:49:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:43 smithi175 ceph-mon[36128]: pgmap v732: 129 pgs: 129 active+clean; 426 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.4 MiB/s wr, 136 op/s 2024-09-13T15:49:44.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57657 16.72 MB/sec execute 80 sec latency 606.071 ms 2024-09-13T15:49:44.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29027 7.60 MB/sec execute 80 sec latency 174.861 ms 2024-09-13T15:49:45.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:44 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:49:45.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:44 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:49:45.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:44 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:49:45.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:44 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:49:45.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:44 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:49:45.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:44 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:49:45.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:44 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:49:45.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:44 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:49:45.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57719 16.53 MB/sec execute 81 sec latency 501.883 ms 2024-09-13T15:49:45.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29141 7.54 MB/sec execute 81 sec latency 170.542 ms 2024-09-13T15:49:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:45 smithi090 ceph-mon[27811]: pgmap v733: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 152 op/s 2024-09-13T15:49:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:45 smithi175 ceph-mon[36128]: pgmap v733: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.5 MiB/s wr, 152 op/s 2024-09-13T15:49:46.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 57950 16.54 MB/sec execute 82 sec latency 513.160 ms 2024-09-13T15:49:46.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29270 7.51 MB/sec execute 82 sec latency 186.672 ms 2024-09-13T15:49:47.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 58179 16.50 MB/sec execute 83 sec latency 739.196 ms 2024-09-13T15:49:47.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29407 7.46 MB/sec execute 83 sec latency 178.287 ms 2024-09-13T15:49:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:47 smithi090 ceph-mon[27811]: pgmap v734: 129 pgs: 129 active+clean; 429 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-09-13T15:49:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:47 smithi175 ceph-mon[36128]: pgmap v734: 129 pgs: 129 active+clean; 429 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-09-13T15:49:48.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 58430 16.42 MB/sec execute 84 sec latency 767.076 ms 2024-09-13T15:49:48.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29568 7.48 MB/sec execute 84 sec latency 113.112 ms 2024-09-13T15:49:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:49:49.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 58781 16.34 MB/sec execute 85 sec latency 758.310 ms 2024-09-13T15:49:49.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29662 7.47 MB/sec execute 85 sec latency 434.938 ms 2024-09-13T15:49:50.017 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:49 smithi175 ceph-mon[36128]: pgmap v735: 129 pgs: 129 active+clean; 426 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 181 op/s 2024-09-13T15:49:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:49 smithi090 ceph-mon[27811]: pgmap v735: 129 pgs: 129 active+clean; 426 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 4.2 MiB/s wr, 181 op/s 2024-09-13T15:49:50.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 58796 16.15 MB/sec execute 86 sec latency 633.768 ms 2024-09-13T15:49:50.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 29825 7.46 MB/sec execute 86 sec latency 113.279 ms 2024-09-13T15:49:51.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 59123 16.10 MB/sec execute 87 sec latency 711.712 ms 2024-09-13T15:49:51.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 30027 7.56 MB/sec execute 87 sec latency 115.695 ms 2024-09-13T15:49:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:51 smithi090 ceph-mon[27811]: pgmap v736: 129 pgs: 129 active+clean; 423 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-09-13T15:49:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:51 smithi175 ceph-mon[36128]: pgmap v736: 129 pgs: 129 active+clean; 423 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.0 MiB/s wr, 140 op/s 2024-09-13T15:49:52.362 INFO:tasks.workunit.client.0.smithi090.stdout: 10 59348 16.12 MB/sec execute 88 sec latency 721.476 ms 2024-09-13T15:49:52.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 30166 7.49 MB/sec execute 88 sec latency 199.944 ms 2024-09-13T15:49:53.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 59580 16.19 MB/sec execute 89 sec latency 713.998 ms 2024-09-13T15:49:53.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 30327 7.44 MB/sec execute 89 sec latency 130.121 ms 2024-09-13T15:49:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:53 smithi090 ceph-mon[27811]: pgmap v737: 129 pgs: 129 active+clean; 410 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.2 MiB/s wr, 173 op/s 2024-09-13T15:49:54.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:53 smithi175 ceph-mon[36128]: pgmap v737: 129 pgs: 129 active+clean; 410 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 3.2 MiB/s wr, 173 op/s 2024-09-13T15:49:54.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 59819 16.22 MB/sec execute 90 sec latency 716.521 ms 2024-09-13T15:49:54.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 30510 7.40 MB/sec execute 90 sec latency 114.919 ms 2024-09-13T15:49:55.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 59865 16.06 MB/sec execute 91 sec latency 894.812 ms 2024-09-13T15:49:55.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 30691 7.39 MB/sec execute 91 sec latency 98.170 ms 2024-09-13T15:49:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:55 smithi090 ceph-mon[27811]: pgmap v738: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.1 MiB/s wr, 215 op/s 2024-09-13T15:49:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:55 smithi175 ceph-mon[36128]: pgmap v738: 129 pgs: 129 active+clean; 409 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.1 MiB/s wr, 215 op/s 2024-09-13T15:49:56.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 60035 15.98 MB/sec execute 92 sec latency 622.362 ms 2024-09-13T15:49:56.398 INFO:tasks.workunit.client.1.smithi175.stdout: 10 30900 7.44 MB/sec execute 92 sec latency 77.765 ms 2024-09-13T15:49:57.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 60360 16.00 MB/sec execute 93 sec latency 713.520 ms 2024-09-13T15:49:57.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31072 7.52 MB/sec execute 93 sec latency 121.023 ms 2024-09-13T15:49:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:57 smithi090 ceph-mon[27811]: pgmap v739: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.5 MiB/s wr, 187 op/s 2024-09-13T15:49:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:57 smithi175 ceph-mon[36128]: pgmap v739: 129 pgs: 129 active+clean; 411 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.5 MiB/s wr, 187 op/s 2024-09-13T15:49:58.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 60736 16.05 MB/sec execute 94 sec latency 523.040 ms 2024-09-13T15:49:58.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31228 7.55 MB/sec execute 94 sec latency 121.294 ms 2024-09-13T15:49:59.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 60920 15.99 MB/sec execute 95 sec latency 746.436 ms 2024-09-13T15:49:59.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31386 7.63 MB/sec execute 95 sec latency 130.381 ms 2024-09-13T15:50:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:49:59 smithi090 ceph-mon[27811]: pgmap v740: 129 pgs: 129 active+clean; 415 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 229 op/s 2024-09-13T15:50:00.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:49:59 smithi175 ceph-mon[36128]: pgmap v740: 129 pgs: 129 active+clean; 415 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.3 MiB/s wr, 229 op/s 2024-09-13T15:50:00.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 61025 15.88 MB/sec execute 96 sec latency 781.053 ms 2024-09-13T15:50:00.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31565 7.73 MB/sec execute 96 sec latency 176.864 ms 2024-09-13T15:50:01.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:00 smithi090 ceph-mon[27811]: overall HEALTH_OK 2024-09-13T15:50:01.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:00 smithi175 ceph-mon[36128]: overall HEALTH_OK 2024-09-13T15:50:01.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 61126 15.74 MB/sec execute 97 sec latency 832.127 ms 2024-09-13T15:50:01.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31771 7.78 MB/sec execute 97 sec latency 94.279 ms 2024-09-13T15:50:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:01 smithi090 ceph-mon[27811]: pgmap v741: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-13T15:50:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:01 smithi175 ceph-mon[36128]: pgmap v741: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-13T15:50:02.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 61583 15.76 MB/sec execute 98 sec latency 117.285 ms 2024-09-13T15:50:02.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31863 7.77 MB/sec execute 98 sec latency 120.687 ms 2024-09-13T15:50:03.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 62947 16.29 MB/sec execute 99 sec latency 124.983 ms 2024-09-13T15:50:03.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 31968 7.77 MB/sec execute 99 sec latency 85.650 ms 2024-09-13T15:50:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:03 smithi090 ceph-mon[27811]: pgmap v742: 129 pgs: 129 active+clean; 425 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-09-13T15:50:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:03 smithi175 ceph-mon[36128]: pgmap v742: 129 pgs: 129 active+clean; 425 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-09-13T15:50:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:04.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 63926 16.81 MB/sec execute 100 sec latency 136.450 ms 2024-09-13T15:50:04.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32113 7.81 MB/sec execute 100 sec latency 81.252 ms 2024-09-13T15:50:05.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64625 17.00 MB/sec execute 101 sec latency 259.766 ms 2024-09-13T15:50:05.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32271 7.83 MB/sec execute 101 sec latency 114.946 ms 2024-09-13T15:50:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:05 smithi090 ceph-mon[27811]: pgmap v743: 129 pgs: 129 active+clean; 429 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 191 op/s 2024-09-13T15:50:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:05 smithi175 ceph-mon[36128]: pgmap v743: 129 pgs: 129 active+clean; 429 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.5 MiB/s wr, 191 op/s 2024-09-13T15:50:06.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64661 16.84 MB/sec execute 102 sec latency 529.819 ms 2024-09-13T15:50:06.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32376 7.80 MB/sec execute 102 sec latency 199.816 ms 2024-09-13T15:50:07.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64670 16.68 MB/sec execute 103 sec latency 603.677 ms 2024-09-13T15:50:07.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32469 7.74 MB/sec execute 103 sec latency 199.987 ms 2024-09-13T15:50:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:07 smithi090 ceph-mon[27811]: pgmap v744: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 146 op/s 2024-09-13T15:50:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:07 smithi175 ceph-mon[36128]: pgmap v744: 129 pgs: 129 active+clean; 430 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.5 MiB/s wr, 146 op/s 2024-09-13T15:50:08.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64700 16.52 MB/sec execute 104 sec latency 601.485 ms 2024-09-13T15:50:08.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32573 7.69 MB/sec execute 104 sec latency 161.825 ms 2024-09-13T15:50:09.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64715 16.37 MB/sec execute 105 sec latency 620.279 ms 2024-09-13T15:50:09.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32675 7.64 MB/sec execute 105 sec latency 148.070 ms 2024-09-13T15:50:10.171 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:09 smithi175 ceph-mon[36128]: pgmap v745: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.7 MiB/s wr, 149 op/s 2024-09-13T15:50:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:09 smithi090 ceph-mon[27811]: pgmap v745: 129 pgs: 129 active+clean; 431 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.7 MiB/s wr, 149 op/s 2024-09-13T15:50:10.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64778 16.23 MB/sec execute 106 sec latency 599.937 ms 2024-09-13T15:50:10.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32783 7.59 MB/sec execute 106 sec latency 265.481 ms 2024-09-13T15:50:11.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 64954 16.20 MB/sec execute 107 sec latency 670.275 ms 2024-09-13T15:50:11.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 32900 7.57 MB/sec execute 107 sec latency 277.811 ms 2024-09-13T15:50:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:11 smithi090 ceph-mon[27811]: pgmap v746: 129 pgs: 129 active+clean; 433 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.1 MiB/s wr, 127 op/s 2024-09-13T15:50:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:11 smithi175 ceph-mon[36128]: pgmap v746: 129 pgs: 129 active+clean; 433 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.1 MiB/s wr, 127 op/s 2024-09-13T15:50:12.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 65041 16.06 MB/sec execute 108 sec latency 664.394 ms 2024-09-13T15:50:12.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33033 7.55 MB/sec execute 108 sec latency 167.950 ms 2024-09-13T15:50:13.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 65378 16.11 MB/sec execute 109 sec latency 501.963 ms 2024-09-13T15:50:13.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33161 7.54 MB/sec execute 109 sec latency 206.562 ms 2024-09-13T15:50:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:13 smithi090 ceph-mon[27811]: pgmap v747: 129 pgs: 129 active+clean; 419 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 150 op/s 2024-09-13T15:50:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:13 smithi175 ceph-mon[36128]: pgmap v747: 129 pgs: 129 active+clean; 419 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 150 op/s 2024-09-13T15:50:14.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 65570 16.05 MB/sec execute 110 sec latency 709.880 ms 2024-09-13T15:50:14.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33309 7.54 MB/sec execute 110 sec latency 180.279 ms 2024-09-13T15:50:15.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 65882 15.96 MB/sec execute 111 sec latency 741.595 ms 2024-09-13T15:50:15.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33462 7.54 MB/sec execute 111 sec latency 147.234 ms 2024-09-13T15:50:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:15 smithi090 ceph-mon[27811]: pgmap v748: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.4 MiB/s wr, 169 op/s 2024-09-13T15:50:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:15 smithi175 ceph-mon[36128]: pgmap v748: 129 pgs: 129 active+clean; 417 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.4 MiB/s wr, 169 op/s 2024-09-13T15:50:16.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 65897 15.82 MB/sec execute 112 sec latency 742.401 ms 2024-09-13T15:50:16.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33600 7.55 MB/sec execute 112 sec latency 288.102 ms 2024-09-13T15:50:17.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 66309 15.87 MB/sec execute 113 sec latency 669.935 ms 2024-09-13T15:50:17.399 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33732 7.53 MB/sec execute 113 sec latency 259.012 ms 2024-09-13T15:50:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:17 smithi090 ceph-mon[27811]: pgmap v749: 129 pgs: 129 active+clean; 411 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-09-13T15:50:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:17 smithi175 ceph-mon[36128]: pgmap v749: 129 pgs: 129 active+clean; 411 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.0 MiB/s wr, 162 op/s 2024-09-13T15:50:18.363 INFO:tasks.workunit.client.0.smithi090.stdout: 10 66483 15.82 MB/sec execute 114 sec latency 797.138 ms 2024-09-13T15:50:18.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 33888 7.49 MB/sec execute 114 sec latency 145.422 ms 2024-09-13T15:50:19.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:19.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:19.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 67828 16.45 MB/sec execute 115 sec latency 125.701 ms 2024-09-13T15:50:19.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 34077 7.45 MB/sec execute 115 sec latency 96.830 ms 2024-09-13T15:50:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:19 smithi090 ceph-mon[27811]: pgmap v750: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-09-13T15:50:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:19 smithi175 ceph-mon[36128]: pgmap v750: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 2.9 MiB/s wr, 207 op/s 2024-09-13T15:50:20.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 68179 16.45 MB/sec execute 116 sec latency 144.205 ms 2024-09-13T15:50:20.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 34282 7.47 MB/sec execute 116 sec latency 69.576 ms 2024-09-13T15:50:21.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 68330 16.34 MB/sec execute 117 sec latency 175.155 ms 2024-09-13T15:50:21.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 34500 7.54 MB/sec execute 117 sec latency 95.622 ms 2024-09-13T15:50:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:21 smithi090 ceph-mon[27811]: pgmap v751: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.4 MiB/s wr, 192 op/s 2024-09-13T15:50:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:21 smithi175 ceph-mon[36128]: pgmap v751: 129 pgs: 129 active+clean; 408 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.4 MiB/s wr, 192 op/s 2024-09-13T15:50:22.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 68994 16.54 MB/sec execute 118 sec latency 248.905 ms 2024-09-13T15:50:22.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 34675 7.57 MB/sec execute 118 sec latency 95.463 ms 2024-09-13T15:50:23.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 70014 16.75 MB/sec execute 119 sec latency 368.095 ms 2024-09-13T15:50:23.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 34862 7.63 MB/sec execute 119 sec latency 84.803 ms 2024-09-13T15:50:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:23 smithi090 ceph-mon[27811]: pgmap v752: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 MiB/s wr, 206 op/s 2024-09-13T15:50:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:23 smithi175 ceph-mon[36128]: pgmap v752: 129 pgs: 129 active+clean; 415 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.8 MiB/s wr, 206 op/s 2024-09-13T15:50:24.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 70419 16.92 MB/sec execute 120 sec latency 719.642 ms 2024-09-13T15:50:24.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 34997 7.67 MB/sec execute 120 sec latency 165.856 ms 2024-09-13T15:50:25.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 70710 16.91 MB/sec execute 121 sec latency 741.758 ms 2024-09-13T15:50:25.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35166 7.74 MB/sec execute 121 sec latency 164.347 ms 2024-09-13T15:50:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:25 smithi090 ceph-mon[27811]: pgmap v753: 129 pgs: 129 active+clean; 422 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-09-13T15:50:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:25 smithi175 ceph-mon[36128]: pgmap v753: 129 pgs: 129 active+clean; 422 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.4 MiB/s wr, 210 op/s 2024-09-13T15:50:26.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 70962 16.89 MB/sec execute 122 sec latency 707.944 ms 2024-09-13T15:50:26.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35322 7.75 MB/sec execute 122 sec latency 165.528 ms 2024-09-13T15:50:27.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71010 16.77 MB/sec execute 123 sec latency 656.973 ms 2024-09-13T15:50:27.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35412 7.75 MB/sec execute 123 sec latency 114.614 ms 2024-09-13T15:50:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:27 smithi090 ceph-mon[27811]: pgmap v754: 129 pgs: 129 active+clean; 428 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.6 MiB/s wr, 163 op/s 2024-09-13T15:50:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:27 smithi175 ceph-mon[36128]: pgmap v754: 129 pgs: 129 active+clean; 428 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.6 MiB/s wr, 163 op/s 2024-09-13T15:50:28.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71107 16.68 MB/sec execute 124 sec latency 667.615 ms 2024-09-13T15:50:28.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35495 7.72 MB/sec execute 124 sec latency 129.508 ms 2024-09-13T15:50:29.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71564 16.74 MB/sec execute 125 sec latency 723.597 ms 2024-09-13T15:50:29.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35594 7.72 MB/sec execute 125 sec latency 121.877 ms 2024-09-13T15:50:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:29 smithi090 ceph-mon[27811]: pgmap v755: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 184 op/s 2024-09-13T15:50:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:29 smithi175 ceph-mon[36128]: pgmap v755: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 184 op/s 2024-09-13T15:50:30.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71661 16.65 MB/sec execute 126 sec latency 670.469 ms 2024-09-13T15:50:30.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35743 7.77 MB/sec execute 126 sec latency 113.794 ms 2024-09-13T15:50:31.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71670 16.52 MB/sec execute 127 sec latency 540.601 ms 2024-09-13T15:50:31.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35844 7.73 MB/sec execute 127 sec latency 199.936 ms 2024-09-13T15:50:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:31 smithi090 ceph-mon[27811]: pgmap v756: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.2 MiB/s wr, 132 op/s 2024-09-13T15:50:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:31 smithi175 ceph-mon[36128]: pgmap v756: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.2 MiB/s wr, 132 op/s 2024-09-13T15:50:32.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71705 16.40 MB/sec execute 128 sec latency 599.501 ms 2024-09-13T15:50:32.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 35944 7.71 MB/sec execute 128 sec latency 216.596 ms 2024-09-13T15:50:33.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71719 16.27 MB/sec execute 129 sec latency 664.748 ms 2024-09-13T15:50:33.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36016 7.66 MB/sec execute 129 sec latency 272.591 ms 2024-09-13T15:50:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:33 smithi090 ceph-mon[27811]: pgmap v757: 129 pgs: 129 active+clean; 432 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.7 MiB/s wr, 144 op/s 2024-09-13T15:50:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:33 smithi175 ceph-mon[36128]: pgmap v757: 129 pgs: 129 active+clean; 432 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.7 MiB/s wr, 144 op/s 2024-09-13T15:50:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:34.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71722 16.15 MB/sec execute 130 sec latency 690.759 ms 2024-09-13T15:50:34.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36140 7.63 MB/sec execute 130 sec latency 95.896 ms 2024-09-13T15:50:35.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71732 16.03 MB/sec execute 131 sec latency 716.802 ms 2024-09-13T15:50:35.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36235 7.58 MB/sec execute 131 sec latency 148.171 ms 2024-09-13T15:50:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:35 smithi090 ceph-mon[27811]: pgmap v758: 129 pgs: 129 active+clean; 433 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.3 MiB/s wr, 139 op/s 2024-09-13T15:50:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:35 smithi175 ceph-mon[36128]: pgmap v758: 129 pgs: 129 active+clean; 433 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.3 MiB/s wr, 139 op/s 2024-09-13T15:50:36.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71776 15.91 MB/sec execute 132 sec latency 541.856 ms 2024-09-13T15:50:36.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36353 7.56 MB/sec execute 132 sec latency 202.962 ms 2024-09-13T15:50:37.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 71875 15.81 MB/sec execute 133 sec latency 570.433 ms 2024-09-13T15:50:37.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36487 7.55 MB/sec execute 133 sec latency 153.823 ms 2024-09-13T15:50:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:37 smithi090 ceph-mon[27811]: pgmap v759: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 117 op/s 2024-09-13T15:50:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:37 smithi175 ceph-mon[36128]: pgmap v759: 129 pgs: 129 active+clean; 430 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.7 MiB/s wr, 117 op/s 2024-09-13T15:50:38.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 72635 16.02 MB/sec execute 134 sec latency 370.292 ms 2024-09-13T15:50:38.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36642 7.54 MB/sec execute 134 sec latency 127.733 ms 2024-09-13T15:50:39.364 INFO:tasks.workunit.client.0.smithi090.stdout: 10 73743 16.27 MB/sec execute 135 sec latency 126.222 ms 2024-09-13T15:50:39.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 36823 7.55 MB/sec execute 135 sec latency 71.840 ms 2024-09-13T15:50:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:39 smithi090 ceph-mon[27811]: pgmap v760: 129 pgs: 129 active+clean; 423 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.6 MiB/s wr, 174 op/s 2024-09-13T15:50:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:39 smithi175 ceph-mon[36128]: pgmap v760: 129 pgs: 129 active+clean; 423 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.6 MiB/s wr, 174 op/s 2024-09-13T15:50:40.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 74687 16.62 MB/sec execute 136 sec latency 136.444 ms 2024-09-13T15:50:40.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 37006 7.56 MB/sec execute 136 sec latency 90.495 ms 2024-09-13T15:50:41.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 75303 16.72 MB/sec execute 137 sec latency 117.692 ms 2024-09-13T15:50:41.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 37200 7.60 MB/sec execute 137 sec latency 79.728 ms 2024-09-13T15:50:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:41 smithi090 ceph-mon[27811]: pgmap v761: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-09-13T15:50:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:41 smithi175 ceph-mon[36128]: pgmap v761: 129 pgs: 129 active+clean; 421 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 2.6 MiB/s wr, 139 op/s 2024-09-13T15:50:42.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 75387 16.62 MB/sec execute 138 sec latency 386.585 ms 2024-09-13T15:50:42.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 37382 7.57 MB/sec execute 138 sec latency 96.886 ms 2024-09-13T15:50:43.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 75416 16.50 MB/sec execute 139 sec latency 584.009 ms 2024-09-13T15:50:43.478 INFO:tasks.workunit.client.1.smithi175.stdout: 10 37519 7.53 MB/sec execute 139 sec latency 193.771 ms 2024-09-13T15:50:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:43 smithi090 ceph-mon[27811]: pgmap v762: 129 pgs: 129 active+clean; 416 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.0 MiB/s wr, 218 op/s 2024-09-13T15:50:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:43 smithi175 ceph-mon[36128]: pgmap v762: 129 pgs: 129 active+clean; 416 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 3.0 MiB/s wr, 218 op/s 2024-09-13T15:50:44.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 75606 16.48 MB/sec execute 140 sec latency 621.188 ms 2024-09-13T15:50:44.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 37666 7.50 MB/sec execute 140 sec latency 135.876 ms 2024-09-13T15:50:45.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:44 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:50:45.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:44 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:50:45.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 75686 16.37 MB/sec execute 141 sec latency 703.926 ms 2024-09-13T15:50:45.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 37838 7.50 MB/sec execute 141 sec latency 127.729 ms 2024-09-13T15:50:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:45 smithi090 ceph-mon[27811]: pgmap v763: 129 pgs: 129 active+clean; 416 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-09-13T15:50:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:50:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:50:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:45 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:50:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:45 smithi175 ceph-mon[36128]: pgmap v763: 129 pgs: 129 active+clean; 416 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.8 MiB/s wr, 219 op/s 2024-09-13T15:50:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:50:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:50:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:45 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:50:46.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 76102 16.44 MB/sec execute 142 sec latency 683.551 ms 2024-09-13T15:50:46.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38041 7.55 MB/sec execute 142 sec latency 135.098 ms 2024-09-13T15:50:47.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 76289 16.37 MB/sec execute 143 sec latency 773.310 ms 2024-09-13T15:50:47.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38175 7.57 MB/sec execute 143 sec latency 162.575 ms 2024-09-13T15:50:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:47 smithi090 ceph-mon[27811]: pgmap v764: 129 pgs: 129 active+clean; 419 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.7 MiB/s wr, 194 op/s 2024-09-13T15:50:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:47 smithi175 ceph-mon[36128]: pgmap v764: 129 pgs: 129 active+clean; 419 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.7 MiB/s wr, 194 op/s 2024-09-13T15:50:48.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 76554 16.29 MB/sec execute 144 sec latency 837.977 ms 2024-09-13T15:50:48.400 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38326 7.59 MB/sec execute 144 sec latency 119.458 ms 2024-09-13T15:50:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:50:49.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 76797 16.22 MB/sec execute 145 sec latency 681.325 ms 2024-09-13T15:50:49.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38448 7.59 MB/sec execute 145 sec latency 174.614 ms 2024-09-13T15:50:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:49 smithi090 ceph-mon[27811]: pgmap v765: 129 pgs: 129 active+clean; 424 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 230 op/s 2024-09-13T15:50:50.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:49 smithi175 ceph-mon[36128]: pgmap v765: 129 pgs: 129 active+clean; 424 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.1 MiB/s wr, 230 op/s 2024-09-13T15:50:50.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 77004 16.23 MB/sec execute 146 sec latency 751.267 ms 2024-09-13T15:50:50.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38678 7.71 MB/sec execute 146 sec latency 110.862 ms 2024-09-13T15:50:51.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 77161 16.18 MB/sec execute 147 sec latency 699.174 ms 2024-09-13T15:50:51.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38858 7.73 MB/sec execute 147 sec latency 149.739 ms 2024-09-13T15:50:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:51 smithi090 ceph-mon[27811]: pgmap v766: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-13T15:50:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:51 smithi175 ceph-mon[36128]: pgmap v766: 129 pgs: 129 active+clean; 422 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 2.8 MiB/s wr, 167 op/s 2024-09-13T15:50:52.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 77465 16.26 MB/sec execute 148 sec latency 668.659 ms 2024-09-13T15:50:52.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 38959 7.75 MB/sec execute 148 sec latency 216.946 ms 2024-09-13T15:50:53.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 77599 16.22 MB/sec execute 149 sec latency 757.175 ms 2024-09-13T15:50:53.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39022 7.71 MB/sec execute 149 sec latency 221.292 ms 2024-09-13T15:50:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:53 smithi090 ceph-mon[27811]: pgmap v767: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-09-13T15:50:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:53 smithi175 ceph-mon[36128]: pgmap v767: 129 pgs: 129 active+clean; 436 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 MiB/s wr, 200 op/s 2024-09-13T15:50:54.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 77757 16.15 MB/sec execute 150 sec latency 578.807 ms 2024-09-13T15:50:54.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39109 7.69 MB/sec execute 150 sec latency 173.588 ms 2024-09-13T15:50:55.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 78110 16.19 MB/sec execute 151 sec latency 575.022 ms 2024-09-13T15:50:55.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39269 7.75 MB/sec execute 151 sec latency 164.969 ms 2024-09-13T15:50:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:55 smithi090 ceph-mon[27811]: pgmap v768: 129 pgs: 129 active+clean; 438 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.0 MiB/s wr, 126 op/s 2024-09-13T15:50:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:55 smithi175 ceph-mon[36128]: pgmap v768: 129 pgs: 129 active+clean; 438 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.0 MiB/s wr, 126 op/s 2024-09-13T15:50:56.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 78817 16.33 MB/sec execute 152 sec latency 489.985 ms 2024-09-13T15:50:56.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39425 7.73 MB/sec execute 152 sec latency 93.217 ms 2024-09-13T15:50:57.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 78931 16.24 MB/sec execute 153 sec latency 111.287 ms 2024-09-13T15:50:57.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39556 7.71 MB/sec execute 153 sec latency 105.688 ms 2024-09-13T15:50:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:57 smithi090 ceph-mon[27811]: pgmap v769: 129 pgs: 129 active+clean; 442 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.4 MiB/s wr, 138 op/s 2024-09-13T15:50:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:57 smithi175 ceph-mon[36128]: pgmap v769: 129 pgs: 129 active+clean; 442 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 4.4 MiB/s wr, 138 op/s 2024-09-13T15:50:58.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 79932 16.45 MB/sec execute 154 sec latency 125.934 ms 2024-09-13T15:50:58.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39677 7.68 MB/sec execute 154 sec latency 105.674 ms 2024-09-13T15:50:59.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 80967 16.75 MB/sec execute 155 sec latency 168.798 ms 2024-09-13T15:50:59.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39795 7.64 MB/sec execute 155 sec latency 109.315 ms 2024-09-13T15:51:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:50:59 smithi090 ceph-mon[27811]: pgmap v770: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 166 op/s 2024-09-13T15:51:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:50:59 smithi175 ceph-mon[36128]: pgmap v770: 129 pgs: 129 active+clean; 441 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.8 MiB/s wr, 166 op/s 2024-09-13T15:51:00.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 81456 16.83 MB/sec execute 156 sec latency 506.202 ms 2024-09-13T15:51:00.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 39931 7.63 MB/sec execute 156 sec latency 165.009 ms 2024-09-13T15:51:01.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 81664 16.79 MB/sec execute 157 sec latency 1030.956 ms 2024-09-13T15:51:01.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40033 7.61 MB/sec execute 157 sec latency 457.062 ms 2024-09-13T15:51:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:01 smithi090 ceph-mon[27811]: pgmap v771: 129 pgs: 129 active+clean; 439 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 122 op/s 2024-09-13T15:51:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:01 smithi175 ceph-mon[36128]: pgmap v771: 129 pgs: 129 active+clean; 439 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.2 MiB/s wr, 122 op/s 2024-09-13T15:51:02.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 81686 16.69 MB/sec execute 158 sec latency 961.421 ms 2024-09-13T15:51:02.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40180 7.61 MB/sec execute 158 sec latency 130.754 ms 2024-09-13T15:51:03.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82224 16.77 MB/sec execute 159 sec latency 825.842 ms 2024-09-13T15:51:03.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40312 7.59 MB/sec execute 159 sec latency 179.911 ms 2024-09-13T15:51:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:03 smithi090 ceph-mon[27811]: pgmap v772: 129 pgs: 129 active+clean; 425 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 146 op/s 2024-09-13T15:51:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:03 smithi175 ceph-mon[36128]: pgmap v772: 129 pgs: 129 active+clean; 425 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.3 MiB/s wr, 146 op/s 2024-09-13T15:51:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:04.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82309 16.70 MB/sec execute 160 sec latency 769.655 ms 2024-09-13T15:51:04.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40462 7.60 MB/sec execute 160 sec latency 134.042 ms 2024-09-13T15:51:05.365 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82334 16.60 MB/sec execute 161 sec latency 527.727 ms 2024-09-13T15:51:05.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40618 7.57 MB/sec execute 161 sec latency 120.367 ms 2024-09-13T15:51:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:05 smithi090 ceph-mon[27811]: pgmap v773: 129 pgs: 129 active+clean; 425 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.5 MiB/s wr, 175 op/s 2024-09-13T15:51:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:05 smithi175 ceph-mon[36128]: pgmap v773: 129 pgs: 129 active+clean; 425 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.5 MiB/s wr, 175 op/s 2024-09-13T15:51:06.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82356 16.50 MB/sec execute 162 sec latency 566.861 ms 2024-09-13T15:51:06.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40782 7.60 MB/sec execute 162 sec latency 144.518 ms 2024-09-13T15:51:07.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82369 16.40 MB/sec execute 163 sec latency 519.395 ms 2024-09-13T15:51:07.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 40936 7.58 MB/sec execute 163 sec latency 116.237 ms 2024-09-13T15:51:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:07 smithi090 ceph-mon[27811]: pgmap v774: 129 pgs: 129 active+clean; 420 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.2 MiB/s wr, 185 op/s 2024-09-13T15:51:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:07 smithi175 ceph-mon[36128]: pgmap v774: 129 pgs: 129 active+clean; 420 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.2 MiB/s wr, 185 op/s 2024-09-13T15:51:08.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82394 16.30 MB/sec execute 164 sec latency 560.544 ms 2024-09-13T15:51:08.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 41102 7.56 MB/sec execute 164 sec latency 98.560 ms 2024-09-13T15:51:09.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82420 16.21 MB/sec execute 165 sec latency 515.602 ms 2024-09-13T15:51:09.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 41238 7.53 MB/sec execute 165 sec latency 118.605 ms 2024-09-13T15:51:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:09 smithi090 ceph-mon[27811]: pgmap v775: 129 pgs: 129 active+clean; 423 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-09-13T15:51:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:09 smithi175 ceph-mon[36128]: pgmap v775: 129 pgs: 129 active+clean; 423 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-09-13T15:51:10.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82435 16.11 MB/sec execute 166 sec latency 681.382 ms 2024-09-13T15:51:10.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 41408 7.55 MB/sec execute 166 sec latency 138.897 ms 2024-09-13T15:51:11.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82466 16.02 MB/sec execute 167 sec latency 685.562 ms 2024-09-13T15:51:11.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 41592 7.57 MB/sec execute 167 sec latency 143.778 ms 2024-09-13T15:51:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:11 smithi090 ceph-mon[27811]: pgmap v776: 129 pgs: 129 active+clean; 423 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.3 MiB/s wr, 176 op/s 2024-09-13T15:51:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:11 smithi175 ceph-mon[36128]: pgmap v776: 129 pgs: 129 active+clean; 423 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.3 MiB/s wr, 176 op/s 2024-09-13T15:51:12.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82480 15.93 MB/sec execute 168 sec latency 714.123 ms 2024-09-13T15:51:12.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 41774 7.60 MB/sec execute 168 sec latency 99.135 ms 2024-09-13T15:51:13.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82544 15.84 MB/sec execute 169 sec latency 578.875 ms 2024-09-13T15:51:13.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 41903 7.61 MB/sec execute 169 sec latency 252.082 ms 2024-09-13T15:51:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:13 smithi090 ceph-mon[27811]: pgmap v777: 129 pgs: 129 active+clean; 424 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-09-13T15:51:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:13 smithi175 ceph-mon[36128]: pgmap v777: 129 pgs: 129 active+clean; 424 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 2.8 MiB/s wr, 197 op/s 2024-09-13T15:51:14.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82725 15.83 MB/sec execute 170 sec latency 525.543 ms 2024-09-13T15:51:14.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42037 7.63 MB/sec execute 170 sec latency 109.774 ms 2024-09-13T15:51:15.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 82814 15.74 MB/sec execute 171 sec latency 610.166 ms 2024-09-13T15:51:15.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42235 7.70 MB/sec execute 171 sec latency 139.997 ms 2024-09-13T15:51:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:15 smithi090 ceph-mon[27811]: pgmap v778: 129 pgs: 129 active+clean; 426 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.2 MiB/s wr, 210 op/s 2024-09-13T15:51:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:15 smithi175 ceph-mon[36128]: pgmap v778: 129 pgs: 129 active+clean; 426 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 3.2 MiB/s wr, 210 op/s 2024-09-13T15:51:16.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 83254 15.80 MB/sec execute 172 sec latency 689.952 ms 2024-09-13T15:51:16.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42408 7.72 MB/sec execute 172 sec latency 187.438 ms 2024-09-13T15:51:17.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 83571 15.77 MB/sec execute 173 sec latency 594.303 ms 2024-09-13T15:51:17.401 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42511 7.73 MB/sec execute 173 sec latency 279.331 ms 2024-09-13T15:51:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:17 smithi090 ceph-mon[27811]: pgmap v779: 129 pgs: 129 active+clean; 433 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.4 MiB/s wr, 152 op/s 2024-09-13T15:51:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:17 smithi175 ceph-mon[36128]: pgmap v779: 129 pgs: 129 active+clean; 433 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.4 MiB/s wr, 152 op/s 2024-09-13T15:51:18.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 83650 15.68 MB/sec execute 174 sec latency 698.113 ms 2024-09-13T15:51:18.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42564 7.70 MB/sec execute 174 sec latency 324.304 ms 2024-09-13T15:51:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:19.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 83930 15.65 MB/sec execute 175 sec latency 716.443 ms 2024-09-13T15:51:19.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42644 7.68 MB/sec execute 175 sec latency 126.261 ms 2024-09-13T15:51:19.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:20 smithi175 ceph-mon[36128]: pgmap v780: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.2 MiB/s wr, 173 op/s 2024-09-13T15:51:20.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 84146 15.65 MB/sec execute 176 sec latency 749.020 ms 2024-09-13T15:51:20.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42758 7.69 MB/sec execute 176 sec latency 123.692 ms 2024-09-13T15:51:20.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:20 smithi090 ceph-mon[27811]: pgmap v780: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.2 MiB/s wr, 173 op/s 2024-09-13T15:51:21.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 84402 15.69 MB/sec execute 177 sec latency 620.906 ms 2024-09-13T15:51:21.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 42898 7.70 MB/sec execute 177 sec latency 149.207 ms 2024-09-13T15:51:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:22 smithi175 ceph-mon[36128]: pgmap v781: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.8 MiB/s wr, 118 op/s 2024-09-13T15:51:22.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 84637 15.70 MB/sec execute 178 sec latency 619.014 ms 2024-09-13T15:51:22.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43011 7.68 MB/sec execute 178 sec latency 172.477 ms 2024-09-13T15:51:22.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:22 smithi090 ceph-mon[27811]: pgmap v781: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 3.8 MiB/s wr, 118 op/s 2024-09-13T15:51:23.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 84724 15.65 MB/sec execute 179 sec latency 782.410 ms 2024-09-13T15:51:23.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43116 7.66 MB/sec execute 179 sec latency 174.322 ms 2024-09-13T15:51:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:24 smithi175 ceph-mon[36128]: pgmap v782: 129 pgs: 129 active+clean; 445 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 4.5 MiB/s wr, 148 op/s 2024-09-13T15:51:24.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 84880 15.61 MB/sec execute 180 sec latency 991.080 ms 2024-09-13T15:51:24.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43230 7.63 MB/sec execute 180 sec latency 175.671 ms 2024-09-13T15:51:24.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:24 smithi090 ceph-mon[27811]: pgmap v782: 129 pgs: 129 active+clean; 445 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.9 KiB/s rd, 4.5 MiB/s wr, 148 op/s 2024-09-13T15:51:25.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 85212 15.62 MB/sec execute 181 sec latency 628.878 ms 2024-09-13T15:51:25.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43345 7.60 MB/sec execute 181 sec latency 101.919 ms 2024-09-13T15:51:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:26 smithi175 ceph-mon[36128]: pgmap v783: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 143 op/s 2024-09-13T15:51:26.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 85933 15.74 MB/sec execute 182 sec latency 368.710 ms 2024-09-13T15:51:26.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43500 7.60 MB/sec execute 182 sec latency 118.374 ms 2024-09-13T15:51:26.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:26 smithi090 ceph-mon[27811]: pgmap v783: 129 pgs: 129 active+clean; 442 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.4 MiB/s wr, 143 op/s 2024-09-13T15:51:27.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 86193 15.70 MB/sec execute 183 sec latency 99.132 ms 2024-09-13T15:51:27.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43654 7.58 MB/sec execute 183 sec latency 108.076 ms 2024-09-13T15:51:28.267 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:28 smithi175 ceph-mon[36128]: pgmap v784: 129 pgs: 129 active+clean; 438 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 118 op/s 2024-09-13T15:51:28.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 87201 15.87 MB/sec execute 184 sec latency 249.881 ms 2024-09-13T15:51:28.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43822 7.60 MB/sec execute 184 sec latency 97.780 ms 2024-09-13T15:51:28.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:28 smithi090 ceph-mon[27811]: pgmap v784: 129 pgs: 129 active+clean; 438 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 118 op/s 2024-09-13T15:51:29.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 88241 16.16 MB/sec execute 185 sec latency 201.440 ms 2024-09-13T15:51:29.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 43991 7.60 MB/sec execute 185 sec latency 124.342 ms 2024-09-13T15:51:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:30 smithi175 ceph-mon[36128]: pgmap v785: 129 pgs: 129 active+clean; 437 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.8 MiB/s wr, 175 op/s 2024-09-13T15:51:30.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 88757 16.23 MB/sec execute 186 sec latency 456.679 ms 2024-09-13T15:51:30.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 44179 7.59 MB/sec execute 186 sec latency 79.868 ms 2024-09-13T15:51:30.448 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:30 smithi090 ceph-mon[27811]: pgmap v785: 129 pgs: 129 active+clean; 437 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 3.8 MiB/s wr, 175 op/s 2024-09-13T15:51:31.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89223 16.28 MB/sec execute 187 sec latency 490.560 ms 2024-09-13T15:51:31.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 44346 7.62 MB/sec execute 187 sec latency 168.886 ms 2024-09-13T15:51:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:32 smithi175 ceph-mon[36128]: pgmap v786: 129 pgs: 129 active+clean; 435 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.9 MiB/s wr, 145 op/s 2024-09-13T15:51:32.366 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89319 16.22 MB/sec execute 188 sec latency 696.258 ms 2024-09-13T15:51:32.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 44478 7.59 MB/sec execute 188 sec latency 226.654 ms 2024-09-13T15:51:32.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:32 smithi090 ceph-mon[27811]: pgmap v786: 129 pgs: 129 active+clean; 435 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.9 MiB/s wr, 145 op/s 2024-09-13T15:51:33.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89424 16.16 MB/sec execute 189 sec latency 538.898 ms 2024-09-13T15:51:33.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 44603 7.57 MB/sec execute 189 sec latency 241.601 ms 2024-09-13T15:51:34.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89427 16.08 MB/sec execute 190 sec latency 728.986 ms 2024-09-13T15:51:34.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 44730 7.54 MB/sec execute 190 sec latency 254.926 ms 2024-09-13T15:51:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:34 smithi090 ceph-mon[27811]: pgmap v787: 129 pgs: 129 active+clean; 426 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-09-13T15:51:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:34 smithi175 ceph-mon[36128]: pgmap v787: 129 pgs: 129 active+clean; 426 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.2 MiB/s wr, 179 op/s 2024-09-13T15:51:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:35.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89452 15.99 MB/sec execute 191 sec latency 586.329 ms 2024-09-13T15:51:35.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 44862 7.53 MB/sec execute 191 sec latency 180.985 ms 2024-09-13T15:51:36.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89477 15.91 MB/sec execute 192 sec latency 600.995 ms 2024-09-13T15:51:36.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45041 7.54 MB/sec execute 192 sec latency 108.403 ms 2024-09-13T15:51:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:36 smithi090 ceph-mon[27811]: pgmap v788: 129 pgs: 129 active+clean; 426 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-09-13T15:51:36.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:36 smithi175 ceph-mon[36128]: pgmap v788: 129 pgs: 129 active+clean; 426 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.0 MiB/s wr, 200 op/s 2024-09-13T15:51:37.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89485 15.83 MB/sec execute 193 sec latency 598.784 ms 2024-09-13T15:51:37.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45246 7.58 MB/sec execute 193 sec latency 107.117 ms 2024-09-13T15:51:38.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89517 15.76 MB/sec execute 194 sec latency 519.062 ms 2024-09-13T15:51:38.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45374 7.58 MB/sec execute 194 sec latency 147.488 ms 2024-09-13T15:51:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:38 smithi090 ceph-mon[27811]: pgmap v789: 129 pgs: 129 active+clean; 425 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-09-13T15:51:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:38 smithi175 ceph-mon[36128]: pgmap v789: 129 pgs: 129 active+clean; 425 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-09-13T15:51:39.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89548 15.68 MB/sec execute 195 sec latency 554.149 ms 2024-09-13T15:51:39.402 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45491 7.60 MB/sec execute 195 sec latency 126.734 ms 2024-09-13T15:51:40.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89557 15.60 MB/sec execute 196 sec latency 648.022 ms 2024-09-13T15:51:40.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45614 7.61 MB/sec execute 196 sec latency 190.838 ms 2024-09-13T15:51:40.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:40 smithi090 ceph-mon[27811]: pgmap v790: 129 pgs: 129 active+clean; 432 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.1 MiB/s wr, 223 op/s 2024-09-13T15:51:40.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:40 smithi175 ceph-mon[36128]: pgmap v790: 129 pgs: 129 active+clean; 432 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.1 MiB/s wr, 223 op/s 2024-09-13T15:51:41.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89576 15.52 MB/sec execute 197 sec latency 654.544 ms 2024-09-13T15:51:41.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45769 7.65 MB/sec execute 197 sec latency 164.949 ms 2024-09-13T15:51:42.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 89861 15.52 MB/sec execute 198 sec latency 440.181 ms 2024-09-13T15:51:42.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 45928 7.66 MB/sec execute 198 sec latency 216.384 ms 2024-09-13T15:51:42.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:42 smithi090 ceph-mon[27811]: pgmap v791: 129 pgs: 129 active+clean; 433 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-09-13T15:51:42.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:42 smithi175 ceph-mon[36128]: pgmap v791: 129 pgs: 129 active+clean; 433 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-09-13T15:51:43.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 91107 15.73 MB/sec execute 199 sec latency 128.525 ms 2024-09-13T15:51:43.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46087 7.69 MB/sec execute 199 sec latency 105.570 ms 2024-09-13T15:51:44.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 92255 16.04 MB/sec execute 200 sec latency 132.055 ms 2024-09-13T15:51:44.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46179 7.68 MB/sec execute 200 sec latency 78.042 ms 2024-09-13T15:51:44.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:44 smithi090 ceph-mon[27811]: pgmap v792: 129 pgs: 129 active+clean; 442 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.7 MiB/s wr, 190 op/s 2024-09-13T15:51:44.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:44 smithi175 ceph-mon[36128]: pgmap v792: 129 pgs: 129 active+clean; 442 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.7 MiB/s wr, 190 op/s 2024-09-13T15:51:45.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 92972 16.15 MB/sec execute 201 sec latency 189.420 ms 2024-09-13T15:51:45.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46299 7.68 MB/sec execute 201 sec latency 84.125 ms 2024-09-13T15:51:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:46 smithi090 ceph-mon[27811]: pgmap v793: 129 pgs: 129 active+clean; 443 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-13T15:51:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:51:46.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 93099 16.08 MB/sec execute 202 sec latency 134.407 ms 2024-09-13T15:51:46.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46457 7.70 MB/sec execute 202 sec latency 125.358 ms 2024-09-13T15:51:46.519 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:46 smithi175 ceph-mon[36128]: pgmap v793: 129 pgs: 129 active+clean; 443 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-13T15:51:46.519 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:51:47.314 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:47 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:51:47.315 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:47 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:51:47.315 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:47 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:51:47.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 93853 16.19 MB/sec execute 203 sec latency 128.336 ms 2024-09-13T15:51:47.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46611 7.68 MB/sec execute 203 sec latency 98.477 ms 2024-09-13T15:51:47.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:47 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:51:47.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:47 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:51:47.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:47 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:51:48.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 94231 16.18 MB/sec execute 204 sec latency 709.254 ms 2024-09-13T15:51:48.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46736 7.67 MB/sec execute 204 sec latency 144.317 ms 2024-09-13T15:51:48.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:48 smithi090 ceph-mon[27811]: pgmap v794: 129 pgs: 129 active+clean; 449 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-09-13T15:51:48.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:48 smithi175 ceph-mon[36128]: pgmap v794: 129 pgs: 129 active+clean; 449 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-09-13T15:51:49.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 94295 16.11 MB/sec execute 205 sec latency 734.501 ms 2024-09-13T15:51:49.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46849 7.64 MB/sec execute 205 sec latency 118.298 ms 2024-09-13T15:51:49.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:49.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:51:50.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 94654 16.12 MB/sec execute 206 sec latency 714.765 ms 2024-09-13T15:51:50.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 46954 7.63 MB/sec execute 206 sec latency 275.677 ms 2024-09-13T15:51:50.416 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:50 smithi175 ceph-mon[36128]: pgmap v795: 129 pgs: 129 active+clean; 449 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 145 op/s 2024-09-13T15:51:50.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:50 smithi090 ceph-mon[27811]: pgmap v795: 129 pgs: 129 active+clean; 449 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 145 op/s 2024-09-13T15:51:51.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 94793 16.08 MB/sec execute 207 sec latency 717.496 ms 2024-09-13T15:51:51.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47085 7.62 MB/sec execute 207 sec latency 169.371 ms 2024-09-13T15:51:52.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 95072 16.12 MB/sec execute 208 sec latency 663.723 ms 2024-09-13T15:51:52.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47208 7.59 MB/sec execute 208 sec latency 177.171 ms 2024-09-13T15:51:52.428 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:52 smithi175 ceph-mon[36128]: pgmap v796: 129 pgs: 129 active+clean; 449 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 108 op/s 2024-09-13T15:51:52.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:52 smithi090 ceph-mon[27811]: pgmap v796: 129 pgs: 129 active+clean; 449 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.8 MiB/s wr, 108 op/s 2024-09-13T15:51:53.367 INFO:tasks.workunit.client.0.smithi090.stdout: 10 95261 16.12 MB/sec execute 209 sec latency 730.973 ms 2024-09-13T15:51:53.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47356 7.60 MB/sec execute 209 sec latency 171.458 ms 2024-09-13T15:51:54.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 95362 16.06 MB/sec execute 210 sec latency 584.618 ms 2024-09-13T15:51:54.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47504 7.58 MB/sec execute 210 sec latency 147.707 ms 2024-09-13T15:51:54.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:54 smithi090 ceph-mon[27811]: pgmap v797: 129 pgs: 129 active+clean; 443 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 183 op/s 2024-09-13T15:51:54.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:54 smithi175 ceph-mon[36128]: pgmap v797: 129 pgs: 129 active+clean; 443 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 183 op/s 2024-09-13T15:51:55.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 95620 16.06 MB/sec execute 211 sec latency 872.498 ms 2024-09-13T15:51:55.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47675 7.59 MB/sec execute 211 sec latency 117.033 ms 2024-09-13T15:51:56.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 95831 16.04 MB/sec execute 212 sec latency 625.198 ms 2024-09-13T15:51:56.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47815 7.59 MB/sec execute 212 sec latency 193.092 ms 2024-09-13T15:51:56.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:56 smithi090 ceph-mon[27811]: pgmap v798: 129 pgs: 129 active+clean; 434 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 168 op/s 2024-09-13T15:51:56.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:56 smithi175 ceph-mon[36128]: pgmap v798: 129 pgs: 129 active+clean; 434 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.7 MiB/s wr, 168 op/s 2024-09-13T15:51:57.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 95958 16.00 MB/sec execute 213 sec latency 631.393 ms 2024-09-13T15:51:57.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 47997 7.59 MB/sec execute 213 sec latency 119.102 ms 2024-09-13T15:51:58.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 96404 16.03 MB/sec execute 214 sec latency 734.082 ms 2024-09-13T15:51:58.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 48167 7.58 MB/sec execute 214 sec latency 121.110 ms 2024-09-13T15:51:58.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:51:58 smithi090 ceph-mon[27811]: pgmap v799: 129 pgs: 129 active+clean; 430 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-09-13T15:51:58.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:51:58 smithi175 ceph-mon[36128]: pgmap v799: 129 pgs: 129 active+clean; 430 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.3 MiB/s wr, 176 op/s 2024-09-13T15:51:59.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 96424 15.96 MB/sec execute 215 sec latency 746.257 ms 2024-09-13T15:51:59.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 48325 7.57 MB/sec execute 215 sec latency 117.089 ms 2024-09-13T15:52:00.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 96568 15.92 MB/sec execute 216 sec latency 507.886 ms 2024-09-13T15:52:00.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 48501 7.57 MB/sec execute 216 sec latency 101.092 ms 2024-09-13T15:52:00.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:00 smithi090 ceph-mon[27811]: pgmap v800: 129 pgs: 129 active+clean; 423 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-09-13T15:52:00.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:00 smithi175 ceph-mon[36128]: pgmap v800: 129 pgs: 129 active+clean; 423 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 2.9 MiB/s wr, 169 op/s 2024-09-13T15:52:01.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 96659 15.85 MB/sec execute 217 sec latency 304.654 ms 2024-09-13T15:52:01.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 48717 7.60 MB/sec execute 217 sec latency 78.416 ms 2024-09-13T15:52:02.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 97536 15.98 MB/sec execute 218 sec latency 135.962 ms 2024-09-13T15:52:02.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 48905 7.62 MB/sec execute 218 sec latency 86.345 ms 2024-09-13T15:52:02.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:02 smithi090 ceph-mon[27811]: pgmap v801: 129 pgs: 129 active+clean; 426 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-09-13T15:52:02.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:02 smithi175 ceph-mon[36128]: pgmap v801: 129 pgs: 129 active+clean; 426 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 197 op/s 2024-09-13T15:52:03.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 98352 16.08 MB/sec execute 219 sec latency 332.403 ms 2024-09-13T15:52:03.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49048 7.64 MB/sec execute 219 sec latency 86.600 ms 2024-09-13T15:52:03.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:03 smithi090 ceph-mon[27811]: pgmap v802: 129 pgs: 129 active+clean; 431 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-13T15:52:03.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:03 smithi175 ceph-mon[36128]: pgmap v802: 129 pgs: 129 active+clean; 431 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-13T15:52:04.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 99378 16.32 MB/sec execute 220 sec latency 134.320 ms 2024-09-13T15:52:04.403 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49231 7.69 MB/sec execute 220 sec latency 78.107 ms 2024-09-13T15:52:04.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:04.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:05.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100060 16.42 MB/sec execute 221 sec latency 457.257 ms 2024-09-13T15:52:05.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49428 7.72 MB/sec execute 221 sec latency 103.778 ms 2024-09-13T15:52:05.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:05 smithi090 ceph-mon[27811]: pgmap v803: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-09-13T15:52:05.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:05 smithi175 ceph-mon[36128]: pgmap v803: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 3.2 MiB/s wr, 178 op/s 2024-09-13T15:52:06.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100068 16.34 MB/sec execute 222 sec latency 650.060 ms 2024-09-13T15:52:06.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49598 7.75 MB/sec execute 222 sec latency 156.106 ms 2024-09-13T15:52:07.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100092 16.27 MB/sec execute 223 sec latency 632.353 ms 2024-09-13T15:52:07.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49664 7.73 MB/sec execute 223 sec latency 199.018 ms 2024-09-13T15:52:08.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:07 smithi175 ceph-mon[36128]: pgmap v804: 129 pgs: 129 active+clean; 446 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-09-13T15:52:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:07 smithi090 ceph-mon[27811]: pgmap v804: 129 pgs: 129 active+clean; 446 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 3.7 MiB/s wr, 162 op/s 2024-09-13T15:52:08.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100113 16.20 MB/sec execute 224 sec latency 608.884 ms 2024-09-13T15:52:08.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49738 7.71 MB/sec execute 224 sec latency 134.103 ms 2024-09-13T15:52:09.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100121 16.13 MB/sec execute 225 sec latency 642.729 ms 2024-09-13T15:52:09.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49843 7.71 MB/sec execute 225 sec latency 162.936 ms 2024-09-13T15:52:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:09 smithi090 ceph-mon[27811]: pgmap v805: 129 pgs: 129 active+clean; 453 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 189 op/s 2024-09-13T15:52:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:09 smithi175 ceph-mon[36128]: pgmap v805: 129 pgs: 129 active+clean; 453 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 189 op/s 2024-09-13T15:52:10.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100134 16.06 MB/sec execute 226 sec latency 558.762 ms 2024-09-13T15:52:10.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 49943 7.70 MB/sec execute 226 sec latency 238.200 ms 2024-09-13T15:52:11.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100168 16.00 MB/sec execute 227 sec latency 632.202 ms 2024-09-13T15:52:11.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50054 7.70 MB/sec execute 227 sec latency 246.239 ms 2024-09-13T15:52:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:11 smithi090 ceph-mon[27811]: pgmap v806: 129 pgs: 129 active+clean; 453 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.3 MiB/s wr, 158 op/s 2024-09-13T15:52:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:11 smithi175 ceph-mon[36128]: pgmap v806: 129 pgs: 129 active+clean; 453 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.3 MiB/s wr, 158 op/s 2024-09-13T15:52:12.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100186 15.93 MB/sec execute 228 sec latency 698.412 ms 2024-09-13T15:52:12.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50185 7.69 MB/sec execute 228 sec latency 138.416 ms 2024-09-13T15:52:13.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100214 15.86 MB/sec execute 229 sec latency 515.736 ms 2024-09-13T15:52:13.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50287 7.67 MB/sec execute 229 sec latency 150.715 ms 2024-09-13T15:52:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:13 smithi090 ceph-mon[27811]: pgmap v807: 129 pgs: 129 active+clean; 448 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.4 MiB/s wr, 129 op/s 2024-09-13T15:52:14.268 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:13 smithi175 ceph-mon[36128]: pgmap v807: 129 pgs: 129 active+clean; 448 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.4 MiB/s wr, 129 op/s 2024-09-13T15:52:14.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100273 15.80 MB/sec execute 230 sec latency 723.217 ms 2024-09-13T15:52:14.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50419 7.64 MB/sec execute 230 sec latency 111.967 ms 2024-09-13T15:52:15.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100449 15.79 MB/sec execute 231 sec latency 559.186 ms 2024-09-13T15:52:15.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50548 7.64 MB/sec execute 231 sec latency 158.042 ms 2024-09-13T15:52:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:15 smithi090 ceph-mon[27811]: pgmap v808: 129 pgs: 129 active+clean; 448 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.5 MiB/s wr, 159 op/s 2024-09-13T15:52:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:15 smithi175 ceph-mon[36128]: pgmap v808: 129 pgs: 129 active+clean; 448 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.5 MiB/s wr, 159 op/s 2024-09-13T15:52:16.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100585 15.74 MB/sec execute 232 sec latency 719.997 ms 2024-09-13T15:52:16.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50685 7.62 MB/sec execute 232 sec latency 131.019 ms 2024-09-13T15:52:17.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 100878 15.75 MB/sec execute 233 sec latency 813.386 ms 2024-09-13T15:52:17.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50824 7.63 MB/sec execute 233 sec latency 212.617 ms 2024-09-13T15:52:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:17 smithi090 ceph-mon[27811]: pgmap v809: 129 pgs: 129 active+clean; 444 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.9 MiB/s wr, 135 op/s 2024-09-13T15:52:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:17 smithi175 ceph-mon[36128]: pgmap v809: 129 pgs: 129 active+clean; 444 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.9 MiB/s wr, 135 op/s 2024-09-13T15:52:18.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 101082 15.71 MB/sec execute 234 sec latency 758.533 ms 2024-09-13T15:52:18.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 50968 7.61 MB/sec execute 234 sec latency 163.927 ms 2024-09-13T15:52:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:19.368 INFO:tasks.workunit.client.0.smithi090.stdout: 10 101825 15.78 MB/sec execute 235 sec latency 504.232 ms 2024-09-13T15:52:19.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 51133 7.61 MB/sec execute 235 sec latency 113.403 ms 2024-09-13T15:52:20.166 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:19 smithi175 ceph-mon[36128]: pgmap v810: 129 pgs: 129 active+clean; 436 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.4 MiB/s wr, 156 op/s 2024-09-13T15:52:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:19 smithi090 ceph-mon[27811]: pgmap v810: 129 pgs: 129 active+clean; 436 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.4 MiB/s wr, 156 op/s 2024-09-13T15:52:20.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 102724 15.99 MB/sec execute 236 sec latency 219.069 ms 2024-09-13T15:52:20.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 51319 7.61 MB/sec execute 236 sec latency 72.339 ms 2024-09-13T15:52:21.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 103609 16.12 MB/sec execute 237 sec latency 227.477 ms 2024-09-13T15:52:21.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 51519 7.63 MB/sec execute 237 sec latency 71.054 ms 2024-09-13T15:52:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:21 smithi090 ceph-mon[27811]: pgmap v811: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-13T15:52:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:21 smithi175 ceph-mon[36128]: pgmap v811: 129 pgs: 129 active+clean; 439 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.8 MiB/s wr, 161 op/s 2024-09-13T15:52:22.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 103700 16.06 MB/sec execute 238 sec latency 226.469 ms 2024-09-13T15:52:22.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 51702 7.62 MB/sec execute 238 sec latency 77.634 ms 2024-09-13T15:52:23.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 103823 16.00 MB/sec execute 239 sec latency 251.707 ms 2024-09-13T15:52:23.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 51887 7.61 MB/sec execute 239 sec latency 77.407 ms 2024-09-13T15:52:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:23 smithi090 ceph-mon[27811]: pgmap v812: 129 pgs: 129 active+clean; 434 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-09-13T15:52:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:23 smithi175 ceph-mon[36128]: pgmap v812: 129 pgs: 129 active+clean; 434 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.3 MiB/s wr, 223 op/s 2024-09-13T15:52:24.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 104160 16.01 MB/sec execute 240 sec latency 454.987 ms 2024-09-13T15:52:24.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52042 7.61 MB/sec execute 240 sec latency 281.045 ms 2024-09-13T15:52:25.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 104478 16.04 MB/sec execute 241 sec latency 612.814 ms 2024-09-13T15:52:25.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52240 7.64 MB/sec execute 241 sec latency 125.898 ms 2024-09-13T15:52:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:25 smithi090 ceph-mon[27811]: pgmap v813: 129 pgs: 129 active+clean; 430 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-09-13T15:52:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:25 smithi175 ceph-mon[36128]: pgmap v813: 129 pgs: 129 active+clean; 430 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 229 op/s 2024-09-13T15:52:26.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 104631 15.98 MB/sec execute 242 sec latency 737.358 ms 2024-09-13T15:52:26.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52381 7.64 MB/sec execute 242 sec latency 216.573 ms 2024-09-13T15:52:27.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 104900 15.95 MB/sec execute 243 sec latency 835.448 ms 2024-09-13T15:52:27.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52500 7.64 MB/sec execute 243 sec latency 390.919 ms 2024-09-13T15:52:28.186 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:27 smithi175 ceph-mon[36128]: pgmap v814: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-09-13T15:52:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:27 smithi090 ceph-mon[27811]: pgmap v814: 129 pgs: 129 active+clean; 435 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.1 MiB/s wr, 191 op/s 2024-09-13T15:52:28.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 104987 15.89 MB/sec execute 244 sec latency 929.939 ms 2024-09-13T15:52:28.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52627 7.66 MB/sec execute 244 sec latency 402.630 ms 2024-09-13T15:52:29.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 105289 15.88 MB/sec execute 245 sec latency 869.451 ms 2024-09-13T15:52:29.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52809 7.69 MB/sec execute 245 sec latency 152.146 ms 2024-09-13T15:52:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:29 smithi090 ceph-mon[27811]: pgmap v815: 129 pgs: 129 active+clean; 442 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.5 MiB/s wr, 220 op/s 2024-09-13T15:52:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:29 smithi175 ceph-mon[36128]: pgmap v815: 129 pgs: 129 active+clean; 442 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.5 MiB/s wr, 220 op/s 2024-09-13T15:52:30.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 105393 15.85 MB/sec execute 246 sec latency 967.423 ms 2024-09-13T15:52:30.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 52972 7.72 MB/sec execute 246 sec latency 297.787 ms 2024-09-13T15:52:31.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 105657 15.88 MB/sec execute 247 sec latency 977.305 ms 2024-09-13T15:52:31.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53143 7.74 MB/sec execute 247 sec latency 118.288 ms 2024-09-13T15:52:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:31 smithi090 ceph-mon[27811]: pgmap v816: 129 pgs: 129 active+clean; 443 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 195 op/s 2024-09-13T15:52:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:31 smithi175 ceph-mon[36128]: pgmap v816: 129 pgs: 129 active+clean; 443 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.3 MiB/s wr, 195 op/s 2024-09-13T15:52:32.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 105813 15.87 MB/sec execute 248 sec latency 823.554 ms 2024-09-13T15:52:32.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53222 7.72 MB/sec execute 248 sec latency 263.556 ms 2024-09-13T15:52:33.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 105935 15.84 MB/sec execute 249 sec latency 879.894 ms 2024-09-13T15:52:33.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53284 7.70 MB/sec execute 249 sec latency 283.171 ms 2024-09-13T15:52:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:33 smithi090 ceph-mon[27811]: pgmap v817: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.1 MiB/s wr, 159 op/s 2024-09-13T15:52:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:33 smithi175 ceph-mon[36128]: pgmap v817: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 4.1 MiB/s wr, 159 op/s 2024-09-13T15:52:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:34.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 106111 15.81 MB/sec execute 250 sec latency 772.132 ms 2024-09-13T15:52:34.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53391 7.70 MB/sec execute 250 sec latency 155.312 ms 2024-09-13T15:52:35.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 106366 15.81 MB/sec execute 251 sec latency 712.868 ms 2024-09-13T15:52:35.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53524 7.71 MB/sec execute 251 sec latency 122.460 ms 2024-09-13T15:52:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:35 smithi090 ceph-mon[27811]: pgmap v818: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 131 op/s 2024-09-13T15:52:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:35 smithi175 ceph-mon[36128]: pgmap v818: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 131 op/s 2024-09-13T15:52:36.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 106582 15.79 MB/sec execute 252 sec latency 739.034 ms 2024-09-13T15:52:36.404 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53659 7.70 MB/sec execute 252 sec latency 127.196 ms 2024-09-13T15:52:37.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 106860 15.79 MB/sec execute 253 sec latency 1021.746 ms 2024-09-13T15:52:37.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53753 7.69 MB/sec execute 253 sec latency 401.414 ms 2024-09-13T15:52:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:37 smithi090 ceph-mon[27811]: pgmap v819: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-09-13T15:52:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:37 smithi175 ceph-mon[36128]: pgmap v819: 129 pgs: 129 active+clean; 452 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-09-13T15:52:38.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 107042 15.77 MB/sec execute 254 sec latency 1242.661 ms 2024-09-13T15:52:38.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53816 7.66 MB/sec execute 254 sec latency 668.492 ms 2024-09-13T15:52:39.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 107060 15.71 MB/sec execute 255 sec latency 562.779 ms 2024-09-13T15:52:39.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 53922 7.64 MB/sec execute 255 sec latency 146.221 ms 2024-09-13T15:52:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:39 smithi090 ceph-mon[27811]: pgmap v820: 129 pgs: 129 active+clean; 455 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.2 MiB/s wr, 213 op/s 2024-09-13T15:52:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:39 smithi175 ceph-mon[36128]: pgmap v820: 129 pgs: 129 active+clean; 455 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.2 MiB/s wr, 213 op/s 2024-09-13T15:52:40.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 107157 15.67 MB/sec execute 256 sec latency 905.822 ms 2024-09-13T15:52:40.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 54039 7.63 MB/sec execute 256 sec latency 234.513 ms 2024-09-13T15:52:41.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 107209 15.61 MB/sec execute 257 sec latency 622.309 ms 2024-09-13T15:52:41.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 54184 7.62 MB/sec execute 257 sec latency 177.703 ms 2024-09-13T15:52:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:41 smithi090 ceph-mon[27811]: pgmap v821: 129 pgs: 129 active+clean; 454 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 187 op/s 2024-09-13T15:52:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:41 smithi175 ceph-mon[36128]: pgmap v821: 129 pgs: 129 active+clean; 454 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 187 op/s 2024-09-13T15:52:42.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 107303 15.56 MB/sec execute 258 sec latency 274.810 ms 2024-09-13T15:52:42.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 54336 7.62 MB/sec execute 258 sec latency 117.759 ms 2024-09-13T15:52:43.369 INFO:tasks.workunit.client.0.smithi090.stdout: 10 107676 15.58 MB/sec execute 259 sec latency 459.687 ms 2024-09-13T15:52:43.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 54520 7.62 MB/sec execute 259 sec latency 93.152 ms 2024-09-13T15:52:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:43 smithi090 ceph-mon[27811]: pgmap v822: 129 pgs: 129 active+clean; 449 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 266 op/s 2024-09-13T15:52:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:43 smithi175 ceph-mon[36128]: pgmap v822: 129 pgs: 129 active+clean; 449 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 4.0 MiB/s wr, 266 op/s 2024-09-13T15:52:44.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 108769 15.69 MB/sec execute 260 sec latency 242.657 ms 2024-09-13T15:52:44.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 54678 7.62 MB/sec execute 260 sec latency 88.525 ms 2024-09-13T15:52:45.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 109473 15.84 MB/sec execute 261 sec latency 254.206 ms 2024-09-13T15:52:45.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 54875 7.62 MB/sec execute 261 sec latency 70.298 ms 2024-09-13T15:52:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:45 smithi090 ceph-mon[27811]: pgmap v823: 129 pgs: 129 active+clean; 448 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.3 MiB/s wr, 299 op/s 2024-09-13T15:52:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:45 smithi175 ceph-mon[36128]: pgmap v823: 129 pgs: 129 active+clean; 448 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 3.3 MiB/s wr, 299 op/s 2024-09-13T15:52:46.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110017 15.90 MB/sec execute 262 sec latency 584.940 ms 2024-09-13T15:52:46.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55055 7.63 MB/sec execute 262 sec latency 114.192 ms 2024-09-13T15:52:47.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:52:47.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:52:47.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:52:47.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:52:47.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:52:47.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:52:47.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:52:47.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:52:47.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110575 15.95 MB/sec execute 263 sec latency 576.180 ms 2024-09-13T15:52:47.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55196 7.61 MB/sec execute 263 sec latency 127.459 ms 2024-09-13T15:52:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:47 smithi090 ceph-mon[27811]: pgmap v824: 129 pgs: 129 active+clean; 442 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.9 MiB/s wr, 298 op/s 2024-09-13T15:52:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:47 smithi175 ceph-mon[36128]: pgmap v824: 129 pgs: 129 active+clean; 442 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 2.9 MiB/s wr, 298 op/s 2024-09-13T15:52:48.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110589 15.90 MB/sec execute 264 sec latency 821.876 ms 2024-09-13T15:52:48.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55366 7.61 MB/sec execute 264 sec latency 132.613 ms 2024-09-13T15:52:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:52:49.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110609 15.84 MB/sec execute 265 sec latency 500.274 ms 2024-09-13T15:52:49.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55517 7.61 MB/sec execute 265 sec latency 124.099 ms 2024-09-13T15:52:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:49 smithi090 ceph-mon[27811]: pgmap v825: 129 pgs: 129 active+clean; 432 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-09-13T15:52:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:49 smithi175 ceph-mon[36128]: pgmap v825: 129 pgs: 129 active+clean; 432 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-09-13T15:52:50.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110707 15.80 MB/sec execute 266 sec latency 493.250 ms 2024-09-13T15:52:50.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55689 7.61 MB/sec execute 266 sec latency 134.167 ms 2024-09-13T15:52:51.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110732 15.74 MB/sec execute 267 sec latency 608.055 ms 2024-09-13T15:52:51.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55838 7.62 MB/sec execute 267 sec latency 165.022 ms 2024-09-13T15:52:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:51 smithi090 ceph-mon[27811]: pgmap v826: 129 pgs: 129 active+clean; 432 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.5 MiB/s wr, 255 op/s 2024-09-13T15:52:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:51 smithi175 ceph-mon[36128]: pgmap v826: 129 pgs: 129 active+clean; 432 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 2.5 MiB/s wr, 255 op/s 2024-09-13T15:52:52.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110742 15.68 MB/sec execute 268 sec latency 576.548 ms 2024-09-13T15:52:52.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 55974 7.62 MB/sec execute 268 sec latency 145.700 ms 2024-09-13T15:52:53.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110766 15.63 MB/sec execute 269 sec latency 586.770 ms 2024-09-13T15:52:53.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56101 7.63 MB/sec execute 269 sec latency 202.576 ms 2024-09-13T15:52:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:53 smithi090 ceph-mon[27811]: pgmap v827: 129 pgs: 129 active+clean; 439 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.1 MiB/s wr, 261 op/s 2024-09-13T15:52:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:53 smithi175 ceph-mon[36128]: pgmap v827: 129 pgs: 129 active+clean; 439 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.1 MiB/s wr, 261 op/s 2024-09-13T15:52:54.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110797 15.57 MB/sec execute 270 sec latency 958.314 ms 2024-09-13T15:52:54.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56232 7.64 MB/sec execute 270 sec latency 251.912 ms 2024-09-13T15:52:55.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110801 15.51 MB/sec execute 271 sec latency 455.601 ms 2024-09-13T15:52:55.405 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56388 7.67 MB/sec execute 271 sec latency 132.994 ms 2024-09-13T15:52:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:55 smithi090 ceph-mon[27811]: pgmap v828: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.3 MiB/s wr, 214 op/s 2024-09-13T15:52:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:55 smithi175 ceph-mon[36128]: pgmap v828: 129 pgs: 129 active+clean; 445 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 3.3 MiB/s wr, 214 op/s 2024-09-13T15:52:56.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110834 15.46 MB/sec execute 272 sec latency 542.064 ms 2024-09-13T15:52:56.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56538 7.68 MB/sec execute 272 sec latency 150.261 ms 2024-09-13T15:52:57.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110890 15.41 MB/sec execute 273 sec latency 482.685 ms 2024-09-13T15:52:57.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56682 7.70 MB/sec execute 273 sec latency 159.915 ms 2024-09-13T15:52:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:57 smithi090 ceph-mon[27811]: pgmap v829: 129 pgs: 129 active+clean; 449 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 178 op/s 2024-09-13T15:52:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:57 smithi175 ceph-mon[36128]: pgmap v829: 129 pgs: 129 active+clean; 449 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 178 op/s 2024-09-13T15:52:58.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 110933 15.36 MB/sec execute 274 sec latency 583.124 ms 2024-09-13T15:52:58.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56765 7.68 MB/sec execute 274 sec latency 183.965 ms 2024-09-13T15:52:59.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 111174 15.35 MB/sec execute 275 sec latency 543.862 ms 2024-09-13T15:52:59.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56849 7.67 MB/sec execute 275 sec latency 127.095 ms 2024-09-13T15:53:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:52:59 smithi090 ceph-mon[27811]: pgmap v830: 129 pgs: 129 active+clean; 451 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-09-13T15:53:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:52:59 smithi175 ceph-mon[36128]: pgmap v830: 129 pgs: 129 active+clean; 451 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 3.9 MiB/s wr, 163 op/s 2024-09-13T15:53:00.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 111266 15.31 MB/sec execute 276 sec latency 792.534 ms 2024-09-13T15:53:00.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 56938 7.67 MB/sec execute 276 sec latency 215.413 ms 2024-09-13T15:53:01.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 111668 15.34 MB/sec execute 277 sec latency 805.364 ms 2024-09-13T15:53:01.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57043 7.66 MB/sec execute 277 sec latency 146.126 ms 2024-09-13T15:53:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:01 smithi090 ceph-mon[27811]: pgmap v831: 129 pgs: 129 active+clean; 455 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.7 MiB/s wr, 118 op/s 2024-09-13T15:53:02.294 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:01 smithi175 ceph-mon[36128]: pgmap v831: 129 pgs: 129 active+clean; 455 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 3.7 MiB/s wr, 118 op/s 2024-09-13T15:53:02.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 112046 15.32 MB/sec execute 278 sec latency 511.476 ms 2024-09-13T15:53:02.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57197 7.67 MB/sec execute 278 sec latency 113.524 ms 2024-09-13T15:53:03.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 113096 15.52 MB/sec execute 279 sec latency 118.150 ms 2024-09-13T15:53:03.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57341 7.66 MB/sec execute 279 sec latency 86.228 ms 2024-09-13T15:53:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:03 smithi090 ceph-mon[27811]: pgmap v832: 129 pgs: 129 active+clean; 456 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 4.2 MiB/s wr, 125 op/s 2024-09-13T15:53:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:03 smithi175 ceph-mon[36128]: pgmap v832: 129 pgs: 129 active+clean; 456 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 2.8 KiB/s rd, 4.2 MiB/s wr, 125 op/s 2024-09-13T15:53:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:04.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 114130 15.67 MB/sec execute 280 sec latency 403.184 ms 2024-09-13T15:53:04.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57470 7.64 MB/sec execute 280 sec latency 103.695 ms 2024-09-13T15:53:05.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 114308 15.64 MB/sec execute 281 sec latency 123.521 ms 2024-09-13T15:53:05.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57610 7.64 MB/sec execute 281 sec latency 125.527 ms 2024-09-13T15:53:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:05 smithi090 ceph-mon[27811]: pgmap v833: 129 pgs: 129 active+clean; 455 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 143 op/s 2024-09-13T15:53:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:05 smithi175 ceph-mon[36128]: pgmap v833: 129 pgs: 129 active+clean; 455 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 143 op/s 2024-09-13T15:53:06.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 114513 15.61 MB/sec execute 282 sec latency 123.209 ms 2024-09-13T15:53:06.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57764 7.63 MB/sec execute 282 sec latency 118.707 ms 2024-09-13T15:53:07.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 115607 15.73 MB/sec execute 283 sec latency 139.573 ms 2024-09-13T15:53:07.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 57931 7.63 MB/sec execute 283 sec latency 113.222 ms 2024-09-13T15:53:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:07 smithi090 ceph-mon[27811]: pgmap v834: 129 pgs: 129 active+clean; 451 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-09-13T15:53:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:07 smithi175 ceph-mon[36128]: pgmap v834: 129 pgs: 129 active+clean; 451 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 119 op/s 2024-09-13T15:53:08.370 INFO:tasks.workunit.client.0.smithi090.stdout: 10 115973 15.75 MB/sec execute 284 sec latency 663.573 ms 2024-09-13T15:53:08.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58062 7.62 MB/sec execute 284 sec latency 221.039 ms 2024-09-13T15:53:09.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 116128 15.73 MB/sec execute 285 sec latency 610.191 ms 2024-09-13T15:53:09.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58206 7.62 MB/sec execute 285 sec latency 132.331 ms 2024-09-13T15:53:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:09 smithi090 ceph-mon[27811]: pgmap v835: 129 pgs: 129 active+clean; 450 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.9 MiB/s wr, 186 op/s 2024-09-13T15:53:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:09 smithi175 ceph-mon[36128]: pgmap v835: 129 pgs: 129 active+clean; 450 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 3.9 MiB/s wr, 186 op/s 2024-09-13T15:53:10.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 116332 15.74 MB/sec execute 286 sec latency 736.404 ms 2024-09-13T15:53:10.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58335 7.61 MB/sec execute 286 sec latency 289.211 ms 2024-09-13T15:53:11.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 116545 15.74 MB/sec execute 287 sec latency 598.526 ms 2024-09-13T15:53:11.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58472 7.60 MB/sec execute 287 sec latency 229.306 ms 2024-09-13T15:53:11.751 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:11 smithi090 ceph-mon[27811]: pgmap v836: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.3 MiB/s wr, 172 op/s 2024-09-13T15:53:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:11 smithi175 ceph-mon[36128]: pgmap v836: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.3 MiB/s wr, 172 op/s 2024-09-13T15:53:12.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 116612 15.69 MB/sec execute 288 sec latency 501.707 ms 2024-09-13T15:53:12.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58635 7.61 MB/sec execute 288 sec latency 142.512 ms 2024-09-13T15:53:13.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117067 15.73 MB/sec execute 289 sec latency 545.325 ms 2024-09-13T15:53:13.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58783 7.59 MB/sec execute 289 sec latency 171.578 ms 2024-09-13T15:53:14.218 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:13 smithi175 ceph-mon[36128]: pgmap v837: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-09-13T15:53:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:13 smithi090 ceph-mon[27811]: pgmap v837: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.3 MiB/s wr, 181 op/s 2024-09-13T15:53:14.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117188 15.70 MB/sec execute 290 sec latency 699.846 ms 2024-09-13T15:53:14.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 58954 7.59 MB/sec execute 290 sec latency 112.836 ms 2024-09-13T15:53:15.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117205 15.64 MB/sec execute 291 sec latency 781.361 ms 2024-09-13T15:53:15.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 59113 7.59 MB/sec execute 291 sec latency 217.473 ms 2024-09-13T15:53:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:15 smithi090 ceph-mon[27811]: pgmap v838: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-09-13T15:53:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:15 smithi175 ceph-mon[36128]: pgmap v838: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.8 KiB/s rd, 3.2 MiB/s wr, 216 op/s 2024-09-13T15:53:16.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117682 15.68 MB/sec execute 292 sec latency 626.841 ms 2024-09-13T15:53:16.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 59265 7.60 MB/sec execute 292 sec latency 221.677 ms 2024-09-13T15:53:17.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117698 15.63 MB/sec execute 293 sec latency 545.069 ms 2024-09-13T15:53:17.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 59439 7.61 MB/sec execute 293 sec latency 126.075 ms 2024-09-13T15:53:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:17 smithi090 ceph-mon[27811]: pgmap v839: 129 pgs: 129 active+clean; 437 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.7 MiB/s wr, 190 op/s 2024-09-13T15:53:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:17 smithi175 ceph-mon[36128]: pgmap v839: 129 pgs: 129 active+clean; 437 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 2.7 MiB/s wr, 190 op/s 2024-09-13T15:53:18.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117796 15.59 MB/sec execute 294 sec latency 542.957 ms 2024-09-13T15:53:18.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 59577 7.61 MB/sec execute 294 sec latency 149.254 ms 2024-09-13T15:53:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:19.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117827 15.54 MB/sec execute 295 sec latency 492.877 ms 2024-09-13T15:53:19.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 59674 7.61 MB/sec execute 295 sec latency 166.376 ms 2024-09-13T15:53:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:19 smithi090 ceph-mon[27811]: pgmap v840: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-09-13T15:53:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:19 smithi175 ceph-mon[36128]: pgmap v840: 129 pgs: 129 active+clean; 439 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.1 MiB/s wr, 225 op/s 2024-09-13T15:53:20.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117843 15.49 MB/sec execute 296 sec latency 603.087 ms 2024-09-13T15:53:20.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 59831 7.63 MB/sec execute 296 sec latency 140.469 ms 2024-09-13T15:53:21.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117855 15.44 MB/sec execute 297 sec latency 576.392 ms 2024-09-13T15:53:21.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60009 7.66 MB/sec execute 297 sec latency 135.967 ms 2024-09-13T15:53:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:21 smithi090 ceph-mon[27811]: pgmap v841: 129 pgs: 129 active+clean; 442 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-09-13T15:53:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:21 smithi175 ceph-mon[36128]: pgmap v841: 129 pgs: 129 active+clean; 442 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.6 MiB/s wr, 153 op/s 2024-09-13T15:53:22.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117895 15.39 MB/sec execute 298 sec latency 647.058 ms 2024-09-13T15:53:22.406 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60163 7.67 MB/sec execute 298 sec latency 154.558 ms 2024-09-13T15:53:23.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 117912 15.34 MB/sec execute 299 sec latency 592.421 ms 2024-09-13T15:53:23.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60293 7.68 MB/sec execute 299 sec latency 125.196 ms 2024-09-13T15:53:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:23 smithi090 ceph-mon[27811]: pgmap v842: 129 pgs: 129 active+clean; 452 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-13T15:53:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:23 smithi175 ceph-mon[36128]: pgmap v842: 129 pgs: 129 active+clean; 452 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.8 MiB/s wr, 164 op/s 2024-09-13T15:53:24.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 118373 15.36 MB/sec execute 300 sec latency 595.983 ms 2024-09-13T15:53:24.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60391 7.67 MB/sec execute 300 sec latency 115.436 ms 2024-09-13T15:53:25.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 119660 15.52 MB/sec execute 301 sec latency 119.158 ms 2024-09-13T15:53:25.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60511 7.67 MB/sec execute 301 sec latency 109.370 ms 2024-09-13T15:53:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:25 smithi090 ceph-mon[27811]: pgmap v843: 129 pgs: 129 active+clean; 456 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-09-13T15:53:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:25 smithi175 ceph-mon[36128]: pgmap v843: 129 pgs: 129 active+clean; 456 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 166 op/s 2024-09-13T15:53:26.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 120735 15.71 MB/sec execute 302 sec latency 134.740 ms 2024-09-13T15:53:26.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60643 7.67 MB/sec execute 302 sec latency 136.686 ms 2024-09-13T15:53:27.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 121371 15.76 MB/sec execute 303 sec latency 176.245 ms 2024-09-13T15:53:27.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60801 7.67 MB/sec execute 303 sec latency 95.681 ms 2024-09-13T15:53:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:27 smithi090 ceph-mon[27811]: pgmap v844: 129 pgs: 129 active+clean; 456 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.0 MiB/s wr, 129 op/s 2024-09-13T15:53:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:27 smithi175 ceph-mon[36128]: pgmap v844: 129 pgs: 129 active+clean; 456 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.3 KiB/s rd, 4.0 MiB/s wr, 129 op/s 2024-09-13T15:53:28.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 121474 15.72 MB/sec execute 304 sec latency 119.778 ms 2024-09-13T15:53:28.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 60932 7.66 MB/sec execute 304 sec latency 111.748 ms 2024-09-13T15:53:29.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 121539 15.67 MB/sec execute 305 sec latency 601.837 ms 2024-09-13T15:53:29.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61021 7.64 MB/sec execute 305 sec latency 200.749 ms 2024-09-13T15:53:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:29 smithi090 ceph-mon[27811]: pgmap v845: 129 pgs: 129 active+clean; 461 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 173 op/s 2024-09-13T15:53:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:29 smithi175 ceph-mon[36128]: pgmap v845: 129 pgs: 129 active+clean; 461 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.6 MiB/s wr, 173 op/s 2024-09-13T15:53:30.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 121671 15.65 MB/sec execute 306 sec latency 496.590 ms 2024-09-13T15:53:30.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61143 7.64 MB/sec execute 306 sec latency 210.377 ms 2024-09-13T15:53:31.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 121849 15.63 MB/sec execute 307 sec latency 831.808 ms 2024-09-13T15:53:31.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61265 7.63 MB/sec execute 307 sec latency 143.248 ms 2024-09-13T15:53:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:31 smithi090 ceph-mon[27811]: pgmap v846: 129 pgs: 129 active+clean; 461 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.1 MiB/s wr, 128 op/s 2024-09-13T15:53:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:31 smithi175 ceph-mon[36128]: pgmap v846: 129 pgs: 129 active+clean; 461 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.1 MiB/s wr, 128 op/s 2024-09-13T15:53:32.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 121923 15.58 MB/sec execute 308 sec latency 774.157 ms 2024-09-13T15:53:32.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61408 7.62 MB/sec execute 308 sec latency 121.081 ms 2024-09-13T15:53:33.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 122301 15.60 MB/sec execute 309 sec latency 860.461 ms 2024-09-13T15:53:33.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61558 7.62 MB/sec execute 309 sec latency 113.147 ms 2024-09-13T15:53:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:33 smithi090 ceph-mon[27811]: pgmap v847: 129 pgs: 129 active+clean; 447 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-09-13T15:53:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:33 smithi175 ceph-mon[36128]: pgmap v847: 129 pgs: 129 active+clean; 447 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 4.3 MiB/s wr, 148 op/s 2024-09-13T15:53:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:34.371 INFO:tasks.workunit.client.0.smithi090.stdout: 10 122573 15.57 MB/sec execute 310 sec latency 629.597 ms 2024-09-13T15:53:34.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61715 7.61 MB/sec execute 310 sec latency 130.339 ms 2024-09-13T15:53:35.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 122639 15.53 MB/sec execute 311 sec latency 767.746 ms 2024-09-13T15:53:35.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 61888 7.61 MB/sec execute 311 sec latency 112.474 ms 2024-09-13T15:53:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:35 smithi090 ceph-mon[27811]: pgmap v848: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 187 op/s 2024-09-13T15:53:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:35 smithi175 ceph-mon[36128]: pgmap v848: 129 pgs: 129 active+clean; 447 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 187 op/s 2024-09-13T15:53:36.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 123078 15.56 MB/sec execute 312 sec latency 645.597 ms 2024-09-13T15:53:36.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62040 7.60 MB/sec execute 312 sec latency 157.115 ms 2024-09-13T15:53:37.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 123164 15.53 MB/sec execute 313 sec latency 787.648 ms 2024-09-13T15:53:37.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62176 7.61 MB/sec execute 313 sec latency 242.641 ms 2024-09-13T15:53:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:37 smithi090 ceph-mon[27811]: pgmap v849: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-09-13T15:53:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:37 smithi175 ceph-mon[36128]: pgmap v849: 129 pgs: 129 active+clean; 440 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 3.3 MiB/s wr, 170 op/s 2024-09-13T15:53:38.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 123248 15.50 MB/sec execute 314 sec latency 982.782 ms 2024-09-13T15:53:38.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62362 7.61 MB/sec execute 314 sec latency 101.068 ms 2024-09-13T15:53:39.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 123595 15.54 MB/sec execute 315 sec latency 689.669 ms 2024-09-13T15:53:39.407 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62516 7.60 MB/sec execute 315 sec latency 117.626 ms 2024-09-13T15:53:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:39 smithi090 ceph-mon[27811]: pgmap v850: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.4 MiB/s wr, 224 op/s 2024-09-13T15:53:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:39 smithi175 ceph-mon[36128]: pgmap v850: 129 pgs: 129 active+clean; 435 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.4 MiB/s wr, 224 op/s 2024-09-13T15:53:40.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 123691 15.51 MB/sec execute 316 sec latency 566.266 ms 2024-09-13T15:53:40.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62679 7.60 MB/sec execute 316 sec latency 167.765 ms 2024-09-13T15:53:41.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 123900 15.49 MB/sec execute 317 sec latency 515.888 ms 2024-09-13T15:53:41.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62838 7.60 MB/sec execute 317 sec latency 174.619 ms 2024-09-13T15:53:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:41 smithi090 ceph-mon[27811]: pgmap v851: 129 pgs: 129 active+clean; 434 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-09-13T15:53:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:41 smithi175 ceph-mon[36128]: pgmap v851: 129 pgs: 129 active+clean; 434 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.5 MiB/s wr, 172 op/s 2024-09-13T15:53:42.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 124267 15.52 MB/sec execute 318 sec latency 792.851 ms 2024-09-13T15:53:42.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 62995 7.61 MB/sec execute 318 sec latency 141.394 ms 2024-09-13T15:53:43.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 124291 15.47 MB/sec execute 319 sec latency 819.520 ms 2024-09-13T15:53:43.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 63159 7.62 MB/sec execute 319 sec latency 170.509 ms 2024-09-13T15:53:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:43 smithi090 ceph-mon[27811]: pgmap v852: 129 pgs: 129 active+clean; 436 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-09-13T15:53:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:43 smithi175 ceph-mon[36128]: pgmap v852: 129 pgs: 129 active+clean; 436 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 MiB/s wr, 214 op/s 2024-09-13T15:53:44.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 124764 15.50 MB/sec execute 320 sec latency 675.908 ms 2024-09-13T15:53:44.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 63291 7.64 MB/sec execute 320 sec latency 105.579 ms 2024-09-13T15:53:45.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 124935 15.48 MB/sec execute 321 sec latency 517.047 ms 2024-09-13T15:53:45.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 63463 7.67 MB/sec execute 321 sec latency 118.608 ms 2024-09-13T15:53:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:45 smithi090 ceph-mon[27811]: pgmap v853: 129 pgs: 129 active+clean; 441 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-09-13T15:53:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:45 smithi175 ceph-mon[36128]: pgmap v853: 129 pgs: 129 active+clean; 441 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 3.2 MiB/s wr, 201 op/s 2024-09-13T15:53:46.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 125145 15.45 MB/sec execute 322 sec latency 125.779 ms 2024-09-13T15:53:46.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 63646 7.68 MB/sec execute 322 sec latency 96.333 ms 2024-09-13T15:53:47.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:46 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:53:47.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:46 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:53:47.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 126197 15.55 MB/sec execute 323 sec latency 119.059 ms 2024-09-13T15:53:47.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 63813 7.69 MB/sec execute 323 sec latency 106.041 ms 2024-09-13T15:53:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:47 smithi090 ceph-mon[27811]: pgmap v854: 129 pgs: 129 active+clean; 448 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.6 MiB/s wr, 186 op/s 2024-09-13T15:53:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:47 smithi175 ceph-mon[36128]: pgmap v854: 129 pgs: 129 active+clean; 448 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.6 MiB/s wr, 186 op/s 2024-09-13T15:53:48.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 127225 15.71 MB/sec execute 324 sec latency 156.753 ms 2024-09-13T15:53:48.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 63915 7.69 MB/sec execute 324 sec latency 139.295 ms 2024-09-13T15:53:49.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:53:49.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128284 15.85 MB/sec execute 325 sec latency 229.088 ms 2024-09-13T15:53:49.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64036 7.69 MB/sec execute 325 sec latency 132.894 ms 2024-09-13T15:53:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:49 smithi090 ceph-mon[27811]: pgmap v855: 129 pgs: 129 active+clean; 451 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-09-13T15:53:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:53:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:53:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:49 smithi175 ceph-mon[36128]: pgmap v855: 129 pgs: 129 active+clean; 451 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 181 op/s 2024-09-13T15:53:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:53:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:53:50.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128342 15.81 MB/sec execute 326 sec latency 360.154 ms 2024-09-13T15:53:50.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64169 7.69 MB/sec execute 326 sec latency 106.671 ms 2024-09-13T15:53:51.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128463 15.78 MB/sec execute 327 sec latency 479.710 ms 2024-09-13T15:53:51.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64304 7.69 MB/sec execute 327 sec latency 122.063 ms 2024-09-13T15:53:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:51 smithi090 ceph-mon[27811]: pgmap v856: 129 pgs: 129 active+clean; 453 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.9 MiB/s wr, 121 op/s 2024-09-13T15:53:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:51 smithi175 ceph-mon[36128]: pgmap v856: 129 pgs: 129 active+clean; 453 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 3.9 MiB/s wr, 121 op/s 2024-09-13T15:53:52.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128479 15.73 MB/sec execute 328 sec latency 567.607 ms 2024-09-13T15:53:52.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64429 7.68 MB/sec execute 328 sec latency 119.198 ms 2024-09-13T15:53:53.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128490 15.68 MB/sec execute 329 sec latency 613.464 ms 2024-09-13T15:53:53.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64545 7.67 MB/sec execute 329 sec latency 143.723 ms 2024-09-13T15:53:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:53 smithi090 ceph-mon[27811]: pgmap v857: 129 pgs: 129 active+clean; 453 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 142 op/s 2024-09-13T15:53:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:53 smithi175 ceph-mon[36128]: pgmap v857: 129 pgs: 129 active+clean; 453 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.4 MiB/s wr, 142 op/s 2024-09-13T15:53:54.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128526 15.64 MB/sec execute 330 sec latency 572.375 ms 2024-09-13T15:53:54.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64658 7.66 MB/sec execute 330 sec latency 140.353 ms 2024-09-13T15:53:55.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128553 15.59 MB/sec execute 331 sec latency 560.438 ms 2024-09-13T15:53:55.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64763 7.65 MB/sec execute 331 sec latency 172.518 ms 2024-09-13T15:53:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:55 smithi090 ceph-mon[27811]: pgmap v858: 129 pgs: 129 active+clean; 453 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 149 op/s 2024-09-13T15:53:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:55 smithi175 ceph-mon[36128]: pgmap v858: 129 pgs: 129 active+clean; 453 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.5 MiB/s wr, 149 op/s 2024-09-13T15:53:56.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128572 15.55 MB/sec execute 332 sec latency 577.350 ms 2024-09-13T15:53:56.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 64889 7.63 MB/sec execute 332 sec latency 224.634 ms 2024-09-13T15:53:57.372 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128638 15.50 MB/sec execute 333 sec latency 543.253 ms 2024-09-13T15:53:57.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65002 7.63 MB/sec execute 333 sec latency 168.238 ms 2024-09-13T15:53:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:57 smithi090 ceph-mon[27811]: pgmap v859: 129 pgs: 129 active+clean; 448 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.2 MiB/s wr, 149 op/s 2024-09-13T15:53:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:57 smithi175 ceph-mon[36128]: pgmap v859: 129 pgs: 129 active+clean; 448 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 4.9 KiB/s rd, 4.2 MiB/s wr, 149 op/s 2024-09-13T15:53:58.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128731 15.47 MB/sec execute 334 sec latency 531.284 ms 2024-09-13T15:53:58.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65170 7.62 MB/sec execute 334 sec latency 144.217 ms 2024-09-13T15:53:59.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 128862 15.45 MB/sec execute 335 sec latency 759.167 ms 2024-09-13T15:53:59.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65278 7.62 MB/sec execute 335 sec latency 231.643 ms 2024-09-13T15:54:00.240 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:53:59 smithi175 ceph-mon[36128]: pgmap v860: 129 pgs: 129 active+clean; 447 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-09-13T15:54:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:53:59 smithi090 ceph-mon[27811]: pgmap v860: 129 pgs: 129 active+clean; 447 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.7 MiB/s wr, 174 op/s 2024-09-13T15:54:00.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 129283 15.48 MB/sec execute 336 sec latency 738.227 ms 2024-09-13T15:54:00.408 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65451 7.62 MB/sec execute 336 sec latency 110.192 ms 2024-09-13T15:54:01.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 129438 15.45 MB/sec execute 337 sec latency 518.036 ms 2024-09-13T15:54:01.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65623 7.63 MB/sec execute 337 sec latency 138.736 ms 2024-09-13T15:54:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:01 smithi090 ceph-mon[27811]: pgmap v861: 129 pgs: 129 active+clean; 443 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.0 MiB/s wr, 165 op/s 2024-09-13T15:54:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:01 smithi175 ceph-mon[36128]: pgmap v861: 129 pgs: 129 active+clean; 443 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.0 MiB/s wr, 165 op/s 2024-09-13T15:54:02.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 129672 15.41 MB/sec execute 338 sec latency 577.232 ms 2024-09-13T15:54:02.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65774 7.62 MB/sec execute 338 sec latency 112.237 ms 2024-09-13T15:54:03.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 130117 15.44 MB/sec execute 339 sec latency 620.815 ms 2024-09-13T15:54:03.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 65924 7.61 MB/sec execute 339 sec latency 168.141 ms 2024-09-13T15:54:04.246 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:03 smithi175 ceph-mon[36128]: pgmap v862: 129 pgs: 129 active+clean; 438 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-09-13T15:54:04.246 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:03 smithi090 ceph-mon[27811]: pgmap v862: 129 pgs: 129 active+clean; 438 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 3.1 MiB/s wr, 194 op/s 2024-09-13T15:54:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:04.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 130274 15.43 MB/sec execute 340 sec latency 465.403 ms 2024-09-13T15:54:04.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 66089 7.61 MB/sec execute 340 sec latency 187.029 ms 2024-09-13T15:54:05.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 130693 15.48 MB/sec execute 341 sec latency 556.490 ms 2024-09-13T15:54:05.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 66273 7.61 MB/sec execute 341 sec latency 197.114 ms 2024-09-13T15:54:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:05 smithi090 ceph-mon[27811]: pgmap v863: 129 pgs: 129 active+clean; 434 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-09-13T15:54:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:05 smithi175 ceph-mon[36128]: pgmap v863: 129 pgs: 129 active+clean; 434 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.1 MiB/s wr, 226 op/s 2024-09-13T15:54:06.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 130881 15.45 MB/sec execute 342 sec latency 424.659 ms 2024-09-13T15:54:06.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 66452 7.62 MB/sec execute 342 sec latency 110.788 ms 2024-09-13T15:54:07.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 131269 15.48 MB/sec execute 343 sec latency 420.172 ms 2024-09-13T15:54:07.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 66591 7.62 MB/sec execute 343 sec latency 105.506 ms 2024-09-13T15:54:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:07 smithi090 ceph-mon[27811]: pgmap v864: 129 pgs: 129 active+clean; 437 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.7 MiB/s wr, 187 op/s 2024-09-13T15:54:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:07 smithi175 ceph-mon[36128]: pgmap v864: 129 pgs: 129 active+clean; 437 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.7 MiB/s wr, 187 op/s 2024-09-13T15:54:08.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 131830 15.53 MB/sec execute 344 sec latency 467.529 ms 2024-09-13T15:54:08.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 66753 7.64 MB/sec execute 344 sec latency 171.076 ms 2024-09-13T15:54:09.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 132028 15.51 MB/sec execute 345 sec latency 191.288 ms 2024-09-13T15:54:09.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 66924 7.66 MB/sec execute 345 sec latency 103.478 ms 2024-09-13T15:54:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:09 smithi090 ceph-mon[27811]: pgmap v865: 129 pgs: 129 active+clean; 439 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-09-13T15:54:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:09 smithi175 ceph-mon[36128]: pgmap v865: 129 pgs: 129 active+clean; 439 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 3.2 MiB/s wr, 196 op/s 2024-09-13T15:54:10.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 132105 15.47 MB/sec execute 346 sec latency 267.689 ms 2024-09-13T15:54:10.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67118 7.69 MB/sec execute 346 sec latency 104.029 ms 2024-09-13T15:54:11.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 132427 15.47 MB/sec execute 347 sec latency 403.457 ms 2024-09-13T15:54:11.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67302 7.71 MB/sec execute 347 sec latency 84.186 ms 2024-09-13T15:54:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:11 smithi090 ceph-mon[27811]: pgmap v866: 129 pgs: 129 active+clean; 442 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.9 MiB/s wr, 153 op/s 2024-09-13T15:54:12.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:11 smithi175 ceph-mon[36128]: pgmap v866: 129 pgs: 129 active+clean; 442 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 2.9 MiB/s wr, 153 op/s 2024-09-13T15:54:12.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 133522 15.55 MB/sec execute 348 sec latency 186.259 ms 2024-09-13T15:54:12.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67413 7.70 MB/sec execute 348 sec latency 117.893 ms 2024-09-13T15:54:13.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 134426 15.69 MB/sec execute 349 sec latency 173.282 ms 2024-09-13T15:54:13.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67511 7.69 MB/sec execute 349 sec latency 101.615 ms 2024-09-13T15:54:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:13 smithi090 ceph-mon[27811]: pgmap v867: 129 pgs: 129 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.2 MiB/s wr, 192 op/s 2024-09-13T15:54:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:13 smithi175 ceph-mon[36128]: pgmap v867: 129 pgs: 129 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 4.2 MiB/s wr, 192 op/s 2024-09-13T15:54:14.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 134683 15.70 MB/sec execute 350 sec latency 443.072 ms 2024-09-13T15:54:14.409 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67632 7.69 MB/sec execute 350 sec latency 116.098 ms 2024-09-13T15:54:15.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 134837 15.68 MB/sec execute 351 sec latency 503.010 ms 2024-09-13T15:54:15.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67759 7.70 MB/sec execute 351 sec latency 158.373 ms 2024-09-13T15:54:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:15 smithi090 ceph-mon[27811]: pgmap v868: 129 pgs: 129 active+clean; 452 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 170 op/s 2024-09-13T15:54:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:15 smithi175 ceph-mon[36128]: pgmap v868: 129 pgs: 129 active+clean; 452 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.4 MiB/s wr, 170 op/s 2024-09-13T15:54:16.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135378 15.72 MB/sec execute 352 sec latency 519.284 ms 2024-09-13T15:54:16.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67872 7.69 MB/sec execute 352 sec latency 217.958 ms 2024-09-13T15:54:17.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135420 15.68 MB/sec execute 353 sec latency 583.656 ms 2024-09-13T15:54:17.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 67995 7.68 MB/sec execute 353 sec latency 147.329 ms 2024-09-13T15:54:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:17 smithi090 ceph-mon[27811]: pgmap v869: 129 pgs: 129 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.4 MiB/s wr, 133 op/s 2024-09-13T15:54:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:17 smithi175 ceph-mon[36128]: pgmap v869: 129 pgs: 129 active+clean; 453 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.4 MiB/s wr, 133 op/s 2024-09-13T15:54:18.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135433 15.64 MB/sec execute 354 sec latency 500.799 ms 2024-09-13T15:54:18.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68115 7.67 MB/sec execute 354 sec latency 137.599 ms 2024-09-13T15:54:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:19.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135555 15.61 MB/sec execute 355 sec latency 414.815 ms 2024-09-13T15:54:19.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68236 7.66 MB/sec execute 355 sec latency 110.885 ms 2024-09-13T15:54:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:19 smithi090 ceph-mon[27811]: pgmap v870: 129 pgs: 129 active+clean; 444 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 166 op/s 2024-09-13T15:54:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:19 smithi175 ceph-mon[36128]: pgmap v870: 129 pgs: 129 active+clean; 444 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 166 op/s 2024-09-13T15:54:20.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135577 15.57 MB/sec execute 356 sec latency 436.125 ms 2024-09-13T15:54:20.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68366 7.65 MB/sec execute 356 sec latency 138.438 ms 2024-09-13T15:54:21.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135617 15.53 MB/sec execute 357 sec latency 451.762 ms 2024-09-13T15:54:21.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68512 7.65 MB/sec execute 357 sec latency 125.073 ms 2024-09-13T15:54:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:22 smithi175 ceph-mon[36128]: pgmap v871: 129 pgs: 129 active+clean; 443 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 150 op/s 2024-09-13T15:54:22.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135637 15.48 MB/sec execute 358 sec latency 599.557 ms 2024-09-13T15:54:22.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68639 7.64 MB/sec execute 358 sec latency 198.269 ms 2024-09-13T15:54:22.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:22 smithi090 ceph-mon[27811]: pgmap v871: 129 pgs: 129 active+clean; 443 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 150 op/s 2024-09-13T15:54:23.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135655 15.44 MB/sec execute 359 sec latency 595.115 ms 2024-09-13T15:54:23.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68775 7.64 MB/sec execute 359 sec latency 306.811 ms 2024-09-13T15:54:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:24 smithi175 ceph-mon[36128]: pgmap v872: 129 pgs: 129 active+clean; 439 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.4 MiB/s wr, 198 op/s 2024-09-13T15:54:24.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135729 15.40 MB/sec execute 360 sec latency 468.501 ms 2024-09-13T15:54:24.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 68913 7.63 MB/sec execute 360 sec latency 173.039 ms 2024-09-13T15:54:24.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:24 smithi090 ceph-mon[27811]: pgmap v872: 129 pgs: 129 active+clean; 439 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 4.4 MiB/s wr, 198 op/s 2024-09-13T15:54:25.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 135971 15.40 MB/sec execute 361 sec latency 556.080 ms 2024-09-13T15:54:25.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 69075 7.63 MB/sec execute 361 sec latency 107.401 ms 2024-09-13T15:54:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:26 smithi175 ceph-mon[36128]: pgmap v873: 129 pgs: 129 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-09-13T15:54:26.373 INFO:tasks.workunit.client.0.smithi090.stdout: 10 136336 15.43 MB/sec execute 362 sec latency 512.738 ms 2024-09-13T15:54:26.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 69243 7.63 MB/sec execute 362 sec latency 105.461 ms 2024-09-13T15:54:26.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:26 smithi090 ceph-mon[27811]: pgmap v873: 129 pgs: 129 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.3 MiB/s wr, 168 op/s 2024-09-13T15:54:27.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 136659 15.40 MB/sec execute 363 sec latency 574.344 ms 2024-09-13T15:54:27.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 69408 7.63 MB/sec execute 363 sec latency 134.545 ms 2024-09-13T15:54:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:28 smithi175 ceph-mon[36128]: pgmap v874: 129 pgs: 129 active+clean; 433 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-09-13T15:54:28.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 136769 15.36 MB/sec execute 364 sec latency 432.952 ms 2024-09-13T15:54:28.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 69559 7.62 MB/sec execute 364 sec latency 154.471 ms 2024-09-13T15:54:28.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:28 smithi090 ceph-mon[27811]: pgmap v874: 129 pgs: 129 active+clean; 433 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 MiB/s wr, 174 op/s 2024-09-13T15:54:29.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 137161 15.37 MB/sec execute 365 sec latency 521.931 ms 2024-09-13T15:54:29.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 69758 7.63 MB/sec execute 365 sec latency 109.583 ms 2024-09-13T15:54:30.202 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:30 smithi090 ceph-mon[27811]: pgmap v875: 129 pgs: 129 active+clean; 432 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 2.8 MiB/s wr, 196 op/s 2024-09-13T15:54:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:30 smithi175 ceph-mon[36128]: pgmap v875: 129 pgs: 129 active+clean; 432 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 2.8 MiB/s wr, 196 op/s 2024-09-13T15:54:30.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 137360 15.38 MB/sec execute 366 sec latency 462.291 ms 2024-09-13T15:54:30.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 69936 7.63 MB/sec execute 366 sec latency 130.640 ms 2024-09-13T15:54:31.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 137776 15.42 MB/sec execute 367 sec latency 435.305 ms 2024-09-13T15:54:31.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70096 7.64 MB/sec execute 367 sec latency 107.070 ms 2024-09-13T15:54:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:32 smithi175 ceph-mon[36128]: pgmap v876: 129 pgs: 129 active+clean; 435 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 MiB/s wr, 178 op/s 2024-09-13T15:54:32.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 138067 15.42 MB/sec execute 368 sec latency 481.994 ms 2024-09-13T15:54:32.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70252 7.65 MB/sec execute 368 sec latency 154.512 ms 2024-09-13T15:54:32.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:32 smithi090 ceph-mon[27811]: pgmap v876: 129 pgs: 129 active+clean; 435 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.5 MiB/s wr, 178 op/s 2024-09-13T15:54:33.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 138360 15.43 MB/sec execute 369 sec latency 413.626 ms 2024-09-13T15:54:33.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70395 7.66 MB/sec execute 369 sec latency 104.598 ms 2024-09-13T15:54:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:34 smithi175 ceph-mon[36128]: pgmap v877: 129 pgs: 129 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.1 MiB/s wr, 199 op/s 2024-09-13T15:54:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:34.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 138922 15.47 MB/sec execute 370 sec latency 480.794 ms 2024-09-13T15:54:34.495 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70559 7.68 MB/sec execute 370 sec latency 80.368 ms 2024-09-13T15:54:34.504 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:34 smithi090 ceph-mon[27811]: pgmap v877: 129 pgs: 129 active+clean; 437 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.1 MiB/s wr, 199 op/s 2024-09-13T15:54:34.504 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:35.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 138967 15.44 MB/sec execute 371 sec latency 473.726 ms 2024-09-13T15:54:35.410 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70719 7.69 MB/sec execute 371 sec latency 131.609 ms 2024-09-13T15:54:35.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:54:35.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:54:35.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:35 smithi175 ceph-mon[36128]: pgmap v878: 129 pgs: 129 active+clean; 441 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-09-13T15:54:35.996 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:54:35.997 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:54:35.997 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:54:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:54:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:54:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:35 smithi090 ceph-mon[27811]: pgmap v878: 129 pgs: 129 active+clean; 441 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.4 MiB/s wr, 167 op/s 2024-09-13T15:54:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:54:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:54:36.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:54:36.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 138979 15.39 MB/sec execute 372 sec latency 505.105 ms 2024-09-13T15:54:36.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70861 7.70 MB/sec execute 372 sec latency 167.780 ms 2024-09-13T15:54:37.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 139099 15.37 MB/sec execute 373 sec latency 562.429 ms 2024-09-13T15:54:37.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 70957 7.69 MB/sec execute 373 sec latency 148.800 ms 2024-09-13T15:54:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:37 smithi090 ceph-mon[27811]: pgmap v879: 129 pgs: 129 active+clean; 442 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 162 op/s 2024-09-13T15:54:38.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:37 smithi175 ceph-mon[36128]: pgmap v879: 129 pgs: 129 active+clean; 442 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.5 MiB/s wr, 162 op/s 2024-09-13T15:54:38.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 139184 15.33 MB/sec execute 374 sec latency 355.612 ms 2024-09-13T15:54:38.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71059 7.69 MB/sec execute 374 sec latency 92.698 ms 2024-09-13T15:54:39.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 139518 15.34 MB/sec execute 375 sec latency 248.191 ms 2024-09-13T15:54:39.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71196 7.69 MB/sec execute 375 sec latency 91.647 ms 2024-09-13T15:54:40.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:39 smithi090 ceph-mon[27811]: pgmap v880: 129 pgs: 129 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 180 op/s 2024-09-13T15:54:40.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:39 smithi175 ceph-mon[36128]: pgmap v880: 129 pgs: 129 active+clean; 449 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.1 MiB/s wr, 180 op/s 2024-09-13T15:54:40.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 140329 15.38 MB/sec execute 376 sec latency 258.913 ms 2024-09-13T15:54:40.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71352 7.70 MB/sec execute 376 sec latency 100.655 ms 2024-09-13T15:54:41.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 141342 15.52 MB/sec execute 377 sec latency 264.621 ms 2024-09-13T15:54:41.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71489 7.69 MB/sec execute 377 sec latency 135.214 ms 2024-09-13T15:54:42.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:41 smithi090 ceph-mon[27811]: pgmap v881: 129 pgs: 129 active+clean; 452 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.0 MiB/s wr, 139 op/s 2024-09-13T15:54:42.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:41 smithi175 ceph-mon[36128]: pgmap v881: 129 pgs: 129 active+clean; 452 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 4.0 MiB/s wr, 139 op/s 2024-09-13T15:54:42.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 141929 15.57 MB/sec execute 378 sec latency 397.940 ms 2024-09-13T15:54:42.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71636 7.69 MB/sec execute 378 sec latency 103.639 ms 2024-09-13T15:54:43.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142511 15.61 MB/sec execute 379 sec latency 515.668 ms 2024-09-13T15:54:43.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71757 7.68 MB/sec execute 379 sec latency 115.756 ms 2024-09-13T15:54:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:43 smithi090 ceph-mon[27811]: pgmap v882: 129 pgs: 129 active+clean; 451 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.6 MiB/s wr, 153 op/s 2024-09-13T15:54:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:43 smithi175 ceph-mon[36128]: pgmap v882: 129 pgs: 129 active+clean; 451 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.6 MiB/s wr, 153 op/s 2024-09-13T15:54:44.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142523 15.57 MB/sec execute 380 sec latency 605.734 ms 2024-09-13T15:54:44.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 71881 7.67 MB/sec execute 380 sec latency 141.188 ms 2024-09-13T15:54:45.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142648 15.55 MB/sec execute 381 sec latency 566.778 ms 2024-09-13T15:54:45.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72028 7.66 MB/sec execute 381 sec latency 107.131 ms 2024-09-13T15:54:46.212 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:45 smithi175 ceph-mon[36128]: pgmap v883: 129 pgs: 129 active+clean; 445 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 163 op/s 2024-09-13T15:54:46.213 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:45 smithi090 ceph-mon[27811]: pgmap v883: 129 pgs: 129 active+clean; 445 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 163 op/s 2024-09-13T15:54:46.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142664 15.51 MB/sec execute 382 sec latency 557.068 ms 2024-09-13T15:54:46.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72190 7.66 MB/sec execute 382 sec latency 124.113 ms 2024-09-13T15:54:47.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142690 15.47 MB/sec execute 383 sec latency 485.819 ms 2024-09-13T15:54:47.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72321 7.66 MB/sec execute 383 sec latency 159.982 ms 2024-09-13T15:54:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:47 smithi090 ceph-mon[27811]: pgmap v884: 129 pgs: 129 active+clean; 443 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 142 op/s 2024-09-13T15:54:48.268 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:47 smithi175 ceph-mon[36128]: pgmap v884: 129 pgs: 129 active+clean; 443 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 142 op/s 2024-09-13T15:54:48.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142717 15.43 MB/sec execute 384 sec latency 512.398 ms 2024-09-13T15:54:48.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72466 7.65 MB/sec execute 384 sec latency 126.230 ms 2024-09-13T15:54:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:54:49.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142739 15.39 MB/sec execute 385 sec latency 543.782 ms 2024-09-13T15:54:49.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72602 7.65 MB/sec execute 385 sec latency 192.354 ms 2024-09-13T15:54:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:49 smithi090 ceph-mon[27811]: pgmap v885: 129 pgs: 129 active+clean; 435 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-09-13T15:54:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:49 smithi175 ceph-mon[36128]: pgmap v885: 129 pgs: 129 active+clean; 435 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 4.0 MiB/s wr, 202 op/s 2024-09-13T15:54:50.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142752 15.35 MB/sec execute 386 sec latency 618.411 ms 2024-09-13T15:54:50.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72775 7.65 MB/sec execute 386 sec latency 148.908 ms 2024-09-13T15:54:51.374 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142865 15.33 MB/sec execute 387 sec latency 563.636 ms 2024-09-13T15:54:51.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 72913 7.64 MB/sec execute 387 sec latency 163.438 ms 2024-09-13T15:54:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:51 smithi090 ceph-mon[27811]: pgmap v886: 129 pgs: 129 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.2 MiB/s wr, 177 op/s 2024-09-13T15:54:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:51 smithi175 ceph-mon[36128]: pgmap v886: 129 pgs: 129 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.2 MiB/s wr, 177 op/s 2024-09-13T15:54:52.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 142959 15.30 MB/sec execute 388 sec latency 679.901 ms 2024-09-13T15:54:52.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73061 7.65 MB/sec execute 388 sec latency 253.187 ms 2024-09-13T15:54:53.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 143088 15.28 MB/sec execute 389 sec latency 726.495 ms 2024-09-13T15:54:53.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73190 7.63 MB/sec execute 389 sec latency 273.812 ms 2024-09-13T15:54:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:53 smithi090 ceph-mon[27811]: pgmap v887: 129 pgs: 129 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 3.3 MiB/s wr, 201 op/s 2024-09-13T15:54:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:53 smithi175 ceph-mon[36128]: pgmap v887: 129 pgs: 129 active+clean; 430 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 9.5 KiB/s rd, 3.3 MiB/s wr, 201 op/s 2024-09-13T15:54:54.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 143434 15.30 MB/sec execute 390 sec latency 631.328 ms 2024-09-13T15:54:54.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73371 7.64 MB/sec execute 390 sec latency 129.468 ms 2024-09-13T15:54:55.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 143643 15.27 MB/sec execute 391 sec latency 707.644 ms 2024-09-13T15:54:55.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73550 7.65 MB/sec execute 391 sec latency 129.167 ms 2024-09-13T15:54:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:55 smithi090 ceph-mon[27811]: pgmap v888: 129 pgs: 129 active+clean; 433 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-09-13T15:54:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:55 smithi175 ceph-mon[36128]: pgmap v888: 129 pgs: 129 active+clean; 433 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 200 op/s 2024-09-13T15:54:56.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 143852 15.24 MB/sec execute 392 sec latency 818.723 ms 2024-09-13T15:54:56.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73702 7.64 MB/sec execute 392 sec latency 153.614 ms 2024-09-13T15:54:57.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 143882 15.21 MB/sec execute 393 sec latency 582.262 ms 2024-09-13T15:54:57.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73860 7.66 MB/sec execute 393 sec latency 184.289 ms 2024-09-13T15:54:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:57 smithi090 ceph-mon[27811]: pgmap v889: 129 pgs: 129 active+clean; 432 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-09-13T15:54:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:57 smithi175 ceph-mon[36128]: pgmap v889: 129 pgs: 129 active+clean; 432 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.5 MiB/s wr, 175 op/s 2024-09-13T15:54:58.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 144332 15.23 MB/sec execute 394 sec latency 775.695 ms 2024-09-13T15:54:58.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 73974 7.66 MB/sec execute 394 sec latency 270.533 ms 2024-09-13T15:54:59.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 144412 15.20 MB/sec execute 395 sec latency 754.234 ms 2024-09-13T15:54:59.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74129 7.68 MB/sec execute 395 sec latency 280.700 ms 2024-09-13T15:55:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:54:59 smithi090 ceph-mon[27811]: pgmap v890: 129 pgs: 129 active+clean; 436 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.2 MiB/s wr, 203 op/s 2024-09-13T15:55:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:54:59 smithi175 ceph-mon[36128]: pgmap v890: 129 pgs: 129 active+clean; 436 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.2 MiB/s wr, 203 op/s 2024-09-13T15:55:00.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 144829 15.26 MB/sec execute 396 sec latency 497.331 ms 2024-09-13T15:55:00.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74297 7.69 MB/sec execute 396 sec latency 133.971 ms 2024-09-13T15:55:01.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 145061 15.24 MB/sec execute 397 sec latency 615.808 ms 2024-09-13T15:55:01.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74450 7.70 MB/sec execute 397 sec latency 162.672 ms 2024-09-13T15:55:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:01 smithi090 ceph-mon[27811]: pgmap v891: 129 pgs: 129 active+clean; 438 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.9 MiB/s wr, 137 op/s 2024-09-13T15:55:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:01 smithi175 ceph-mon[36128]: pgmap v891: 129 pgs: 129 active+clean; 438 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 2.9 MiB/s wr, 137 op/s 2024-09-13T15:55:02.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 145307 15.24 MB/sec execute 398 sec latency 540.152 ms 2024-09-13T15:55:02.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74542 7.70 MB/sec execute 398 sec latency 270.662 ms 2024-09-13T15:55:03.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 145451 15.23 MB/sec execute 399 sec latency 957.711 ms 2024-09-13T15:55:03.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74609 7.68 MB/sec execute 399 sec latency 398.946 ms 2024-09-13T15:55:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:03 smithi090 ceph-mon[27811]: pgmap v892: 129 pgs: 129 active+clean; 447 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 154 op/s 2024-09-13T15:55:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:03 smithi175 ceph-mon[36128]: pgmap v892: 129 pgs: 129 active+clean; 447 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 154 op/s 2024-09-13T15:55:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:04.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146013 15.27 MB/sec execute 400 sec latency 464.517 ms 2024-09-13T15:55:04.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74701 7.68 MB/sec execute 400 sec latency 118.248 ms 2024-09-13T15:55:05.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146049 15.23 MB/sec execute 401 sec latency 467.789 ms 2024-09-13T15:55:05.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74812 7.67 MB/sec execute 401 sec latency 130.421 ms 2024-09-13T15:55:06.252 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:05 smithi090 ceph-mon[27811]: pgmap v893: 129 pgs: 129 active+clean; 447 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 146 op/s 2024-09-13T15:55:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:05 smithi175 ceph-mon[36128]: pgmap v893: 129 pgs: 129 active+clean; 447 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.1 MiB/s wr, 146 op/s 2024-09-13T15:55:06.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146071 15.20 MB/sec execute 402 sec latency 459.630 ms 2024-09-13T15:55:06.411 INFO:tasks.workunit.client.1.smithi175.stdout: 10 74954 7.68 MB/sec execute 402 sec latency 136.366 ms 2024-09-13T15:55:07.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146203 15.17 MB/sec execute 403 sec latency 475.730 ms 2024-09-13T15:55:07.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75069 7.67 MB/sec execute 403 sec latency 145.535 ms 2024-09-13T15:55:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:07 smithi090 ceph-mon[27811]: pgmap v894: 129 pgs: 129 active+clean; 446 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.9 MiB/s wr, 114 op/s 2024-09-13T15:55:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:07 smithi175 ceph-mon[36128]: pgmap v894: 129 pgs: 129 active+clean; 446 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.9 MiB/s wr, 114 op/s 2024-09-13T15:55:08.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146212 15.14 MB/sec execute 404 sec latency 590.250 ms 2024-09-13T15:55:08.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75176 7.67 MB/sec execute 404 sec latency 236.021 ms 2024-09-13T15:55:09.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146216 15.10 MB/sec execute 405 sec latency 583.410 ms 2024-09-13T15:55:09.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75281 7.66 MB/sec execute 405 sec latency 123.681 ms 2024-09-13T15:55:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:09 smithi090 ceph-mon[27811]: pgmap v895: 129 pgs: 129 active+clean; 446 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-09-13T15:55:10.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:09 smithi175 ceph-mon[36128]: pgmap v895: 129 pgs: 129 active+clean; 446 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 158 op/s 2024-09-13T15:55:10.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146273 15.07 MB/sec execute 406 sec latency 400.774 ms 2024-09-13T15:55:10.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75406 7.65 MB/sec execute 406 sec latency 133.685 ms 2024-09-13T15:55:11.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146295 15.03 MB/sec execute 407 sec latency 406.946 ms 2024-09-13T15:55:11.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75540 7.64 MB/sec execute 407 sec latency 108.518 ms 2024-09-13T15:55:12.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:11 smithi175 ceph-mon[36128]: pgmap v896: 129 pgs: 129 active+clean; 446 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.6 MiB/s wr, 115 op/s 2024-09-13T15:55:12.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:11 smithi090 ceph-mon[27811]: pgmap v896: 129 pgs: 129 active+clean; 446 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.6 MiB/s wr, 115 op/s 2024-09-13T15:55:12.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146356 15.00 MB/sec execute 408 sec latency 526.144 ms 2024-09-13T15:55:12.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75662 7.64 MB/sec execute 408 sec latency 261.496 ms 2024-09-13T15:55:13.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146572 14.99 MB/sec execute 409 sec latency 452.779 ms 2024-09-13T15:55:13.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75809 7.62 MB/sec execute 409 sec latency 143.752 ms 2024-09-13T15:55:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:13 smithi090 ceph-mon[27811]: pgmap v897: 129 pgs: 129 active+clean; 443 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.1 MiB/s wr, 173 op/s 2024-09-13T15:55:14.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:13 smithi175 ceph-mon[36128]: pgmap v897: 129 pgs: 129 active+clean; 443 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.5 KiB/s rd, 4.1 MiB/s wr, 173 op/s 2024-09-13T15:55:14.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 146925 15.01 MB/sec execute 410 sec latency 460.399 ms 2024-09-13T15:55:14.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 75967 7.63 MB/sec execute 410 sec latency 93.956 ms 2024-09-13T15:55:15.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 147192 14.99 MB/sec execute 411 sec latency 484.902 ms 2024-09-13T15:55:15.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76111 7.62 MB/sec execute 411 sec latency 167.101 ms 2024-09-13T15:55:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:15 smithi090 ceph-mon[27811]: pgmap v898: 129 pgs: 129 active+clean; 437 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.2 MiB/s wr, 165 op/s 2024-09-13T15:55:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:15 smithi175 ceph-mon[36128]: pgmap v898: 129 pgs: 129 active+clean; 437 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.2 MiB/s wr, 165 op/s 2024-09-13T15:55:16.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 147397 14.97 MB/sec execute 412 sec latency 796.139 ms 2024-09-13T15:55:16.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76241 7.62 MB/sec execute 412 sec latency 216.529 ms 2024-09-13T15:55:17.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 147717 14.97 MB/sec execute 413 sec latency 615.189 ms 2024-09-13T15:55:17.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76397 7.62 MB/sec execute 413 sec latency 205.287 ms 2024-09-13T15:55:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:17 smithi090 ceph-mon[27811]: pgmap v899: 129 pgs: 129 active+clean; 431 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.9 MiB/s wr, 164 op/s 2024-09-13T15:55:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:17 smithi175 ceph-mon[36128]: pgmap v899: 129 pgs: 129 active+clean; 431 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.9 MiB/s wr, 164 op/s 2024-09-13T15:55:18.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 147927 14.96 MB/sec execute 414 sec latency 650.506 ms 2024-09-13T15:55:18.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76563 7.62 MB/sec execute 414 sec latency 242.319 ms 2024-09-13T15:55:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:19.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 148185 14.98 MB/sec execute 415 sec latency 715.513 ms 2024-09-13T15:55:19.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76709 7.61 MB/sec execute 415 sec latency 159.495 ms 2024-09-13T15:55:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:19 smithi090 ceph-mon[27811]: pgmap v900: 129 pgs: 129 active+clean; 431 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-09-13T15:55:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:19 smithi175 ceph-mon[36128]: pgmap v900: 129 pgs: 129 active+clean; 431 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.1 MiB/s wr, 216 op/s 2024-09-13T15:55:20.375 INFO:tasks.workunit.client.0.smithi090.stdout: 10 148388 14.98 MB/sec execute 416 sec latency 871.139 ms 2024-09-13T15:55:20.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76849 7.61 MB/sec execute 416 sec latency 170.528 ms 2024-09-13T15:55:21.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 148585 14.97 MB/sec execute 417 sec latency 531.227 ms 2024-09-13T15:55:21.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 76976 7.60 MB/sec execute 417 sec latency 132.142 ms 2024-09-13T15:55:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:21 smithi090 ceph-mon[27811]: pgmap v901: 129 pgs: 129 active+clean; 428 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-09-13T15:55:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:21 smithi175 ceph-mon[36128]: pgmap v901: 129 pgs: 129 active+clean; 428 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 6.6 KiB/s rd, 2.5 MiB/s wr, 166 op/s 2024-09-13T15:55:22.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 148857 14.97 MB/sec execute 418 sec latency 551.039 ms 2024-09-13T15:55:22.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 77157 7.61 MB/sec execute 418 sec latency 135.403 ms 2024-09-13T15:55:23.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 148971 14.95 MB/sec execute 419 sec latency 774.087 ms 2024-09-13T15:55:23.412 INFO:tasks.workunit.client.1.smithi175.stdout: 10 77311 7.62 MB/sec execute 419 sec latency 204.460 ms 2024-09-13T15:55:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:23 smithi090 ceph-mon[27811]: pgmap v902: 129 pgs: 129 active+clean; 433 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.9 MiB/s wr, 197 op/s 2024-09-13T15:55:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:23 smithi175 ceph-mon[36128]: pgmap v902: 129 pgs: 129 active+clean; 433 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 2.9 MiB/s wr, 197 op/s 2024-09-13T15:55:24.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149462 14.98 MB/sec execute 420 sec latency 723.941 ms 2024-09-13T15:55:24.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 77440 7.62 MB/sec execute 420 sec latency 113.132 ms 2024-09-13T15:55:25.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149566 14.96 MB/sec execute 421 sec latency 546.029 ms 2024-09-13T15:55:25.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 77565 7.63 MB/sec execute 421 sec latency 177.496 ms 2024-09-13T15:55:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:25 smithi090 ceph-mon[27811]: pgmap v903: 129 pgs: 129 active+clean; 432 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-09-13T15:55:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:25 smithi175 ceph-mon[36128]: pgmap v903: 129 pgs: 129 active+clean; 432 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 2.7 MiB/s wr, 160 op/s 2024-09-13T15:55:26.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149594 14.93 MB/sec execute 422 sec latency 494.690 ms 2024-09-13T15:55:26.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 77702 7.65 MB/sec execute 422 sec latency 169.705 ms 2024-09-13T15:55:27.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149633 14.90 MB/sec execute 423 sec latency 646.716 ms 2024-09-13T15:55:27.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 77842 7.65 MB/sec execute 423 sec latency 302.442 ms 2024-09-13T15:55:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:27 smithi090 ceph-mon[27811]: pgmap v904: 129 pgs: 129 active+clean; 437 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.1 MiB/s wr, 153 op/s 2024-09-13T15:55:28.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:27 smithi175 ceph-mon[36128]: pgmap v904: 129 pgs: 129 active+clean; 437 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.0 KiB/s rd, 3.1 MiB/s wr, 153 op/s 2024-09-13T15:55:28.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149644 14.86 MB/sec execute 424 sec latency 618.134 ms 2024-09-13T15:55:28.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78005 7.66 MB/sec execute 424 sec latency 194.081 ms 2024-09-13T15:55:29.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149747 14.84 MB/sec execute 425 sec latency 524.036 ms 2024-09-13T15:55:29.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78112 7.66 MB/sec execute 425 sec latency 160.609 ms 2024-09-13T15:55:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:29 smithi090 ceph-mon[27811]: pgmap v905: 129 pgs: 129 active+clean; 446 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 174 op/s 2024-09-13T15:55:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:29 smithi175 ceph-mon[36128]: pgmap v905: 129 pgs: 129 active+clean; 446 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.8 MiB/s wr, 174 op/s 2024-09-13T15:55:30.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149784 14.81 MB/sec execute 426 sec latency 557.453 ms 2024-09-13T15:55:30.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78180 7.65 MB/sec execute 426 sec latency 156.205 ms 2024-09-13T15:55:31.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149794 14.77 MB/sec execute 427 sec latency 542.377 ms 2024-09-13T15:55:31.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78249 7.63 MB/sec execute 427 sec latency 195.146 ms 2024-09-13T15:55:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:31 smithi090 ceph-mon[27811]: pgmap v906: 129 pgs: 129 active+clean; 447 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 115 op/s 2024-09-13T15:55:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:31 smithi175 ceph-mon[36128]: pgmap v906: 129 pgs: 129 active+clean; 447 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 115 op/s 2024-09-13T15:55:32.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149815 14.74 MB/sec execute 428 sec latency 605.757 ms 2024-09-13T15:55:32.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78355 7.64 MB/sec execute 428 sec latency 176.972 ms 2024-09-13T15:55:33.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149834 14.70 MB/sec execute 429 sec latency 554.336 ms 2024-09-13T15:55:33.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78491 7.64 MB/sec execute 429 sec latency 140.868 ms 2024-09-13T15:55:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:33 smithi090 ceph-mon[27811]: pgmap v907: 129 pgs: 129 active+clean; 452 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 155 op/s 2024-09-13T15:55:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:33 smithi175 ceph-mon[36128]: pgmap v907: 129 pgs: 129 active+clean; 452 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.3 MiB/s wr, 155 op/s 2024-09-13T15:55:34.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:34.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 149896 14.67 MB/sec execute 430 sec latency 517.503 ms 2024-09-13T15:55:34.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78609 7.63 MB/sec execute 430 sec latency 180.172 ms 2024-09-13T15:55:35.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 150032 14.66 MB/sec execute 431 sec latency 521.303 ms 2024-09-13T15:55:35.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78736 7.63 MB/sec execute 431 sec latency 163.481 ms 2024-09-13T15:55:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:35 smithi090 ceph-mon[27811]: pgmap v908: 129 pgs: 129 active+clean; 447 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 127 op/s 2024-09-13T15:55:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:55:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:55:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:55:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:35 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:55:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:35 smithi175 ceph-mon[36128]: pgmap v908: 129 pgs: 129 active+clean; 447 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.9 MiB/s wr, 127 op/s 2024-09-13T15:55:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:55:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:55:36.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:55:36.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:35 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:55:36.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 150152 14.64 MB/sec execute 432 sec latency 659.975 ms 2024-09-13T15:55:36.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78826 7.62 MB/sec execute 432 sec latency 164.923 ms 2024-09-13T15:55:37.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 150541 14.66 MB/sec execute 433 sec latency 710.965 ms 2024-09-13T15:55:37.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 78938 7.62 MB/sec execute 433 sec latency 340.265 ms 2024-09-13T15:55:38.267 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:38 smithi175 ceph-mon[36128]: pgmap v909: 129 pgs: 129 active+clean; 443 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.8 MiB/s wr, 116 op/s 2024-09-13T15:55:38.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 150636 14.63 MB/sec execute 434 sec latency 817.238 ms 2024-09-13T15:55:38.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79045 7.60 MB/sec execute 434 sec latency 209.999 ms 2024-09-13T15:55:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:38 smithi090 ceph-mon[27811]: pgmap v909: 129 pgs: 129 active+clean; 443 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.8 MiB/s wr, 116 op/s 2024-09-13T15:55:39.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 150868 14.61 MB/sec execute 435 sec latency 657.446 ms 2024-09-13T15:55:39.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79174 7.59 MB/sec execute 435 sec latency 185.457 ms 2024-09-13T15:55:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:40 smithi175 ceph-mon[36128]: pgmap v910: 129 pgs: 129 active+clean; 445 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.8 MiB/s wr, 161 op/s 2024-09-13T15:55:40.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 151190 14.60 MB/sec execute 436 sec latency 892.053 ms 2024-09-13T15:55:40.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79299 7.59 MB/sec execute 436 sec latency 299.398 ms 2024-09-13T15:55:40.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:40 smithi090 ceph-mon[27811]: pgmap v910: 129 pgs: 129 active+clean; 445 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.8 MiB/s wr, 161 op/s 2024-09-13T15:55:41.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 151293 14.59 MB/sec execute 437 sec latency 593.038 ms 2024-09-13T15:55:41.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79428 7.59 MB/sec execute 437 sec latency 253.930 ms 2024-09-13T15:55:42.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:42 smithi175 ceph-mon[36128]: pgmap v911: 129 pgs: 129 active+clean; 444 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 125 op/s 2024-09-13T15:55:42.356 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:42 smithi090 ceph-mon[27811]: pgmap v911: 129 pgs: 129 active+clean; 444 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.9 MiB/s wr, 125 op/s 2024-09-13T15:55:42.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 151666 14.63 MB/sec execute 438 sec latency 496.405 ms 2024-09-13T15:55:42.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79567 7.58 MB/sec execute 438 sec latency 118.668 ms 2024-09-13T15:55:43.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 151913 14.63 MB/sec execute 439 sec latency 586.456 ms 2024-09-13T15:55:43.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79726 7.58 MB/sec execute 439 sec latency 121.012 ms 2024-09-13T15:55:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:44 smithi175 ceph-mon[36128]: pgmap v912: 129 pgs: 129 active+clean; 436 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-09-13T15:55:44.376 INFO:tasks.workunit.client.0.smithi090.stdout: 10 152059 14.61 MB/sec execute 440 sec latency 694.153 ms 2024-09-13T15:55:44.413 INFO:tasks.workunit.client.1.smithi175.stdout: 10 79899 7.58 MB/sec execute 440 sec latency 134.460 ms 2024-09-13T15:55:44.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:44 smithi090 ceph-mon[27811]: pgmap v912: 129 pgs: 129 active+clean; 436 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.4 MiB/s wr, 197 op/s 2024-09-13T15:55:45.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 152184 14.60 MB/sec execute 441 sec latency 785.272 ms 2024-09-13T15:55:45.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80045 7.59 MB/sec execute 441 sec latency 183.633 ms 2024-09-13T15:55:46.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:46 smithi175 ceph-mon[36128]: pgmap v913: 129 pgs: 129 active+clean; 424 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.6 MiB/s wr, 163 op/s 2024-09-13T15:55:46.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 152400 14.59 MB/sec execute 442 sec latency 883.968 ms 2024-09-13T15:55:46.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80228 7.58 MB/sec execute 442 sec latency 113.367 ms 2024-09-13T15:55:46.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:46 smithi090 ceph-mon[27811]: pgmap v913: 129 pgs: 129 active+clean; 424 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.6 MiB/s wr, 163 op/s 2024-09-13T15:55:47.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 152421 14.56 MB/sec execute 443 sec latency 883.978 ms 2024-09-13T15:55:47.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80347 7.58 MB/sec execute 443 sec latency 333.131 ms 2024-09-13T15:55:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:48 smithi175 ceph-mon[36128]: pgmap v914: 129 pgs: 129 active+clean; 425 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.7 MiB/s wr, 173 op/s 2024-09-13T15:55:48.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 152963 14.59 MB/sec execute 444 sec latency 851.303 ms 2024-09-13T15:55:48.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80497 7.57 MB/sec execute 444 sec latency 129.256 ms 2024-09-13T15:55:48.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:48 smithi090 ceph-mon[27811]: pgmap v914: 129 pgs: 129 active+clean; 425 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.1 KiB/s rd, 2.7 MiB/s wr, 173 op/s 2024-09-13T15:55:49.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153101 14.58 MB/sec execute 445 sec latency 753.714 ms 2024-09-13T15:55:49.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80644 7.58 MB/sec execute 445 sec latency 170.917 ms 2024-09-13T15:55:49.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:49.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:55:50.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153121 14.55 MB/sec execute 446 sec latency 575.043 ms 2024-09-13T15:55:50.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80786 7.58 MB/sec execute 446 sec latency 176.933 ms 2024-09-13T15:55:50.416 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:50 smithi175 ceph-mon[36128]: pgmap v915: 129 pgs: 129 active+clean; 429 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.8 MiB/s wr, 211 op/s 2024-09-13T15:55:50.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:50 smithi090 ceph-mon[27811]: pgmap v915: 129 pgs: 129 active+clean; 429 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 8.8 KiB/s rd, 2.8 MiB/s wr, 211 op/s 2024-09-13T15:55:51.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153176 14.52 MB/sec execute 447 sec latency 626.481 ms 2024-09-13T15:55:51.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 80934 7.59 MB/sec execute 447 sec latency 274.783 ms 2024-09-13T15:55:52.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153188 14.49 MB/sec execute 448 sec latency 639.181 ms 2024-09-13T15:55:52.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81048 7.58 MB/sec execute 448 sec latency 134.183 ms 2024-09-13T15:55:52.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:52 smithi090 ceph-mon[27811]: pgmap v916: 129 pgs: 129 active+clean; 431 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-09-13T15:55:52.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:52 smithi175 ceph-mon[36128]: pgmap v916: 129 pgs: 129 active+clean; 431 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5.0 KiB/s rd, 2.3 MiB/s wr, 160 op/s 2024-09-13T15:55:53.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153296 14.47 MB/sec execute 449 sec latency 587.344 ms 2024-09-13T15:55:53.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81196 7.60 MB/sec execute 449 sec latency 130.463 ms 2024-09-13T15:55:54.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153304 14.44 MB/sec execute 450 sec latency 533.061 ms 2024-09-13T15:55:54.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81358 7.61 MB/sec execute 450 sec latency 132.805 ms 2024-09-13T15:55:54.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:54 smithi090 ceph-mon[27811]: pgmap v917: 129 pgs: 129 active+clean; 440 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.3 MiB/s wr, 191 op/s 2024-09-13T15:55:54.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:54 smithi175 ceph-mon[36128]: pgmap v917: 129 pgs: 129 active+clean; 440 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.3 MiB/s wr, 191 op/s 2024-09-13T15:55:55.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153336 14.41 MB/sec execute 451 sec latency 531.449 ms 2024-09-13T15:55:55.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81527 7.62 MB/sec execute 451 sec latency 171.340 ms 2024-09-13T15:55:56.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153360 14.38 MB/sec execute 452 sec latency 551.275 ms 2024-09-13T15:55:56.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81646 7.62 MB/sec execute 452 sec latency 140.149 ms 2024-09-13T15:55:56.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:56 smithi090 ceph-mon[27811]: pgmap v918: 129 pgs: 129 active+clean; 438 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-09-13T15:55:56.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:56 smithi175 ceph-mon[36128]: pgmap v918: 129 pgs: 129 active+clean; 438 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.1 MiB/s wr, 136 op/s 2024-09-13T15:55:57.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 153376 14.35 MB/sec execute 453 sec latency 523.900 ms 2024-09-13T15:55:57.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81731 7.62 MB/sec execute 453 sec latency 124.170 ms 2024-09-13T15:55:58.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 154076 14.40 MB/sec execute 454 sec latency 475.452 ms 2024-09-13T15:55:58.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81843 7.62 MB/sec execute 454 sec latency 101.007 ms 2024-09-13T15:55:58.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:55:58 smithi090 ceph-mon[27811]: pgmap v919: 129 pgs: 129 active+clean; 444 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.5 MiB/s wr, 133 op/s 2024-09-13T15:55:58.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:55:58 smithi175 ceph-mon[36128]: pgmap v919: 129 pgs: 129 active+clean; 444 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.5 MiB/s wr, 133 op/s 2024-09-13T15:55:59.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 154969 14.45 MB/sec execute 455 sec latency 153.866 ms 2024-09-13T15:55:59.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 81998 7.62 MB/sec execute 455 sec latency 76.777 ms 2024-09-13T15:56:00.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 155966 14.56 MB/sec execute 456 sec latency 134.309 ms 2024-09-13T15:56:00.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82160 7.62 MB/sec execute 456 sec latency 109.212 ms 2024-09-13T15:56:00.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:00 smithi090 ceph-mon[27811]: pgmap v920: 129 pgs: 129 active+clean; 450 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 163 op/s 2024-09-13T15:56:00.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:00 smithi175 ceph-mon[36128]: pgmap v920: 129 pgs: 129 active+clean; 450 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 163 op/s 2024-09-13T15:56:01.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 156748 14.63 MB/sec execute 457 sec latency 128.414 ms 2024-09-13T15:56:01.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82297 7.62 MB/sec execute 457 sec latency 112.463 ms 2024-09-13T15:56:02.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 156918 14.61 MB/sec execute 458 sec latency 264.300 ms 2024-09-13T15:56:02.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82419 7.61 MB/sec execute 458 sec latency 137.277 ms 2024-09-13T15:56:02.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:02 smithi090 ceph-mon[27811]: pgmap v921: 129 pgs: 129 active+clean; 450 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.9 MiB/s wr, 123 op/s 2024-09-13T15:56:02.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:02 smithi175 ceph-mon[36128]: pgmap v921: 129 pgs: 129 active+clean; 450 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.9 MiB/s wr, 123 op/s 2024-09-13T15:56:03.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 157189 14.60 MB/sec execute 459 sec latency 411.659 ms 2024-09-13T15:56:03.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82548 7.60 MB/sec execute 459 sec latency 133.748 ms 2024-09-13T15:56:04.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 157634 14.63 MB/sec execute 460 sec latency 604.833 ms 2024-09-13T15:56:04.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82680 7.59 MB/sec execute 460 sec latency 127.899 ms 2024-09-13T15:56:04.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:04 smithi090 ceph-mon[27811]: pgmap v922: 129 pgs: 129 active+clean; 442 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 147 op/s 2024-09-13T15:56:04.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:04 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:04.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:04 smithi175 ceph-mon[36128]: pgmap v922: 129 pgs: 129 active+clean; 442 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.5 MiB/s wr, 147 op/s 2024-09-13T15:56:04.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:04 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:05.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 157941 14.61 MB/sec execute 461 sec latency 711.360 ms 2024-09-13T15:56:05.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82810 7.59 MB/sec execute 461 sec latency 206.407 ms 2024-09-13T15:56:06.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 157990 14.58 MB/sec execute 462 sec latency 501.097 ms 2024-09-13T15:56:06.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 82950 7.59 MB/sec execute 462 sec latency 198.789 ms 2024-09-13T15:56:06.503 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:06 smithi090 ceph-mon[27811]: pgmap v923: 129 pgs: 129 active+clean; 444 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 155 op/s 2024-09-13T15:56:06.527 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:06 smithi175 ceph-mon[36128]: pgmap v923: 129 pgs: 129 active+clean; 444 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.0 MiB/s wr, 155 op/s 2024-09-13T15:56:07.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 158339 14.59 MB/sec execute 463 sec latency 802.048 ms 2024-09-13T15:56:07.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83089 7.58 MB/sec execute 463 sec latency 148.693 ms 2024-09-13T15:56:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:08 smithi090 ceph-mon[27811]: pgmap v924: 129 pgs: 129 active+clean; 437 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 150 op/s 2024-09-13T15:56:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:08 smithi175 ceph-mon[36128]: pgmap v924: 129 pgs: 129 active+clean; 437 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.2 KiB/s rd, 3.6 MiB/s wr, 150 op/s 2024-09-13T15:56:08.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 158479 14.57 MB/sec execute 464 sec latency 796.319 ms 2024-09-13T15:56:08.414 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83212 7.57 MB/sec execute 464 sec latency 210.580 ms 2024-09-13T15:56:09.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 158623 14.57 MB/sec execute 465 sec latency 827.122 ms 2024-09-13T15:56:09.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83341 7.57 MB/sec execute 465 sec latency 268.985 ms 2024-09-13T15:56:10.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 158921 14.60 MB/sec execute 466 sec latency 845.419 ms 2024-09-13T15:56:10.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83476 7.58 MB/sec execute 466 sec latency 172.326 ms 2024-09-13T15:56:10.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:10 smithi090 ceph-mon[27811]: pgmap v925: 129 pgs: 129 active+clean; 429 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-09-13T15:56:10.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:10 smithi175 ceph-mon[36128]: pgmap v925: 129 pgs: 129 active+clean; 429 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.6 MiB/s wr, 210 op/s 2024-09-13T15:56:11.377 INFO:tasks.workunit.client.0.smithi090.stdout: 10 159057 14.58 MB/sec execute 467 sec latency 564.686 ms 2024-09-13T15:56:11.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83646 7.58 MB/sec execute 467 sec latency 107.463 ms 2024-09-13T15:56:12.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 159447 14.59 MB/sec execute 468 sec latency 720.892 ms 2024-09-13T15:56:12.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83783 7.57 MB/sec execute 468 sec latency 264.634 ms 2024-09-13T15:56:12.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:12 smithi090 ceph-mon[27811]: pgmap v926: 129 pgs: 129 active+clean; 429 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.8 MiB/s wr, 171 op/s 2024-09-13T15:56:12.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:12 smithi175 ceph-mon[36128]: pgmap v926: 129 pgs: 129 active+clean; 429 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 7.6 KiB/s rd, 2.8 MiB/s wr, 171 op/s 2024-09-13T15:56:13.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 159491 14.56 MB/sec execute 469 sec latency 649.673 ms 2024-09-13T15:56:13.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 83958 7.57 MB/sec execute 469 sec latency 112.013 ms 2024-09-13T15:56:14.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 159696 14.56 MB/sec execute 470 sec latency 616.503 ms 2024-09-13T15:56:14.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 84128 7.57 MB/sec execute 470 sec latency 149.928 ms 2024-09-13T15:56:14.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:14 smithi090 ceph-mon[27811]: pgmap v927: 129 pgs: 129 active+clean; 426 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-09-13T15:56:14.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:14 smithi175 ceph-mon[36128]: pgmap v927: 129 pgs: 129 active+clean; 426 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 9.2 KiB/s rd, 3.2 MiB/s wr, 198 op/s 2024-09-13T15:56:15.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 160101 14.58 MB/sec execute 471 sec latency 724.937 ms 2024-09-13T15:56:15.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 84300 7.58 MB/sec execute 471 sec latency 154.938 ms 2024-09-13T15:56:16.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 160400 14.58 MB/sec execute 472 sec latency 153.860 ms 2024-09-13T15:56:16.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 84501 7.59 MB/sec execute 472 sec latency 78.260 ms 2024-09-13T15:56:16.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:16 smithi090 ceph-mon[27811]: pgmap v928: 129 pgs: 129 active+clean; 429 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.7 MiB/s wr, 198 op/s 2024-09-13T15:56:16.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:16 smithi175 ceph-mon[36128]: pgmap v928: 129 pgs: 129 active+clean; 429 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.7 MiB/s wr, 198 op/s 2024-09-13T15:56:17.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 160698 14.57 MB/sec execute 473 sec latency 117.653 ms 2024-09-13T15:56:17.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 84636 7.60 MB/sec execute 473 sec latency 119.673 ms 2024-09-13T15:56:18.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 161639 14.63 MB/sec execute 474 sec latency 256.389 ms 2024-09-13T15:56:18.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 84809 7.61 MB/sec execute 474 sec latency 99.157 ms 2024-09-13T15:56:18.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:18 smithi090 ceph-mon[27811]: pgmap v929: 129 pgs: 129 active+clean; 433 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-09-13T15:56:18.519 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:18 smithi175 ceph-mon[36128]: pgmap v929: 129 pgs: 129 active+clean; 433 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.2 KiB/s rd, 2.7 MiB/s wr, 169 op/s 2024-09-13T15:56:19.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 162382 14.70 MB/sec execute 475 sec latency 484.831 ms 2024-09-13T15:56:19.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85033 7.64 MB/sec execute 475 sec latency 92.403 ms 2024-09-13T15:56:19.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:19 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:19.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:19 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:20.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 162732 14.72 MB/sec execute 476 sec latency 495.138 ms 2024-09-13T15:56:20.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85198 7.64 MB/sec execute 476 sec latency 86.940 ms 2024-09-13T15:56:20.416 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:20 smithi175 ceph-mon[36128]: pgmap v930: 129 pgs: 129 active+clean; 441 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-09-13T15:56:20.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:20 smithi090 ceph-mon[27811]: pgmap v930: 129 pgs: 129 active+clean; 441 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.6 MiB/s wr, 199 op/s 2024-09-13T15:56:21.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163006 14.72 MB/sec execute 477 sec latency 621.059 ms 2024-09-13T15:56:21.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85265 7.63 MB/sec execute 477 sec latency 228.526 ms 2024-09-13T15:56:22.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163142 14.70 MB/sec execute 478 sec latency 606.212 ms 2024-09-13T15:56:22.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85338 7.62 MB/sec execute 478 sec latency 158.151 ms 2024-09-13T15:56:22.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:22 smithi090 ceph-mon[27811]: pgmap v931: 129 pgs: 129 active+clean; 442 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.2 MiB/s wr, 135 op/s 2024-09-13T15:56:22.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:22 smithi175 ceph-mon[36128]: pgmap v931: 129 pgs: 129 active+clean; 442 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.2 MiB/s wr, 135 op/s 2024-09-13T15:56:23.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163598 14.72 MB/sec execute 479 sec latency 808.926 ms 2024-09-13T15:56:23.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85465 7.62 MB/sec execute 479 sec latency 130.546 ms 2024-09-13T15:56:24.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163744 14.71 MB/sec execute 480 sec latency 661.347 ms 2024-09-13T15:56:24.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85626 7.63 MB/sec execute 480 sec latency 137.071 ms 2024-09-13T15:56:24.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:24 smithi090 ceph-mon[27811]: pgmap v932: 129 pgs: 129 active+clean; 452 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.6 MiB/s wr, 179 op/s 2024-09-13T15:56:24.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:24 smithi175 ceph-mon[36128]: pgmap v932: 129 pgs: 129 active+clean; 452 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.6 MiB/s wr, 179 op/s 2024-09-13T15:56:25.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163756 14.68 MB/sec execute 481 sec latency 766.326 ms 2024-09-13T15:56:25.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85748 7.63 MB/sec execute 481 sec latency 177.610 ms 2024-09-13T15:56:26.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163815 14.66 MB/sec execute 482 sec latency 598.504 ms 2024-09-13T15:56:26.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85862 7.62 MB/sec execute 482 sec latency 121.456 ms 2024-09-13T15:56:26.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:26 smithi090 ceph-mon[27811]: pgmap v933: 129 pgs: 129 active+clean; 447 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.2 MiB/s wr, 145 op/s 2024-09-13T15:56:26.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:26 smithi175 ceph-mon[36128]: pgmap v933: 129 pgs: 129 active+clean; 447 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.2 MiB/s wr, 145 op/s 2024-09-13T15:56:27.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163823 14.63 MB/sec execute 483 sec latency 535.920 ms 2024-09-13T15:56:27.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 85959 7.61 MB/sec execute 483 sec latency 170.685 ms 2024-09-13T15:56:28.365 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:28 smithi175 ceph-mon[36128]: pgmap v934: 129 pgs: 129 active+clean; 447 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.5 MiB/s wr, 145 op/s 2024-09-13T15:56:28.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163840 14.60 MB/sec execute 484 sec latency 566.656 ms 2024-09-13T15:56:28.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86083 7.60 MB/sec execute 484 sec latency 187.618 ms 2024-09-13T15:56:28.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:28 smithi090 ceph-mon[27811]: pgmap v934: 129 pgs: 129 active+clean; 447 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 4.2 KiB/s rd, 4.5 MiB/s wr, 145 op/s 2024-09-13T15:56:29.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163940 14.58 MB/sec execute 485 sec latency 612.755 ms 2024-09-13T15:56:29.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86207 7.59 MB/sec execute 485 sec latency 151.329 ms 2024-09-13T15:56:30.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163970 14.55 MB/sec execute 486 sec latency 545.921 ms 2024-09-13T15:56:30.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86359 7.59 MB/sec execute 486 sec latency 163.274 ms 2024-09-13T15:56:30.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:30 smithi090 ceph-mon[27811]: pgmap v935: 129 pgs: 129 active+clean; 442 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 180 op/s 2024-09-13T15:56:30.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:30 smithi175 ceph-mon[36128]: pgmap v935: 129 pgs: 129 active+clean; 442 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.9 KiB/s rd, 4.5 MiB/s wr, 180 op/s 2024-09-13T15:56:31.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 163994 14.52 MB/sec execute 487 sec latency 578.550 ms 2024-09-13T15:56:31.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86494 7.59 MB/sec execute 487 sec latency 188.512 ms 2024-09-13T15:56:32.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 164017 14.49 MB/sec execute 488 sec latency 507.722 ms 2024-09-13T15:56:32.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86624 7.58 MB/sec execute 488 sec latency 164.486 ms 2024-09-13T15:56:32.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:32 smithi090 ceph-mon[27811]: pgmap v936: 129 pgs: 129 active+clean; 441 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.6 MiB/s wr, 139 op/s 2024-09-13T15:56:32.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:32 smithi175 ceph-mon[36128]: pgmap v936: 129 pgs: 129 active+clean; 441 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 3.6 MiB/s wr, 139 op/s 2024-09-13T15:56:33.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 164072 14.47 MB/sec execute 489 sec latency 550.068 ms 2024-09-13T15:56:33.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86765 7.58 MB/sec execute 489 sec latency 210.206 ms 2024-09-13T15:56:34.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 165054 14.53 MB/sec execute 490 sec latency 238.545 ms 2024-09-13T15:56:34.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 86981 7.60 MB/sec execute 490 sec latency 76.449 ms 2024-09-13T15:56:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:34 smithi090 ceph-mon[27811]: pgmap v937: 129 pgs: 129 active+clean; 429 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-09-13T15:56:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:34 smithi175 ceph-mon[36128]: pgmap v937: 129 pgs: 129 active+clean; 429 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 3.9 MiB/s wr, 174 op/s 2024-09-13T15:56:34.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:35.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 166130 14.64 MB/sec execute 491 sec latency 134.168 ms 2024-09-13T15:56:35.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 87163 7.59 MB/sec execute 491 sec latency 85.100 ms 2024-09-13T15:56:36.378 INFO:tasks.workunit.client.0.smithi090.stdout: 10 167300 14.74 MB/sec execute 492 sec latency 126.667 ms 2024-09-13T15:56:36.415 INFO:tasks.workunit.client.1.smithi175.stdout: 10 87364 7.60 MB/sec execute 492 sec latency 78.668 ms 2024-09-13T15:56:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:36 smithi090 ceph-mon[27811]: pgmap v938: 129 pgs: 129 active+clean; 430 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-09-13T15:56:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:56:36.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:36 smithi175 ceph-mon[36128]: pgmap v938: 129 pgs: 129 active+clean; 430 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.9 MiB/s wr, 174 op/s 2024-09-13T15:56:36.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:36 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:56:37.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 167519 14.73 MB/sec execute 493 sec latency 114.600 ms 2024-09-13T15:56:37.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 87530 7.59 MB/sec execute 493 sec latency 84.211 ms 2024-09-13T15:56:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:56:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:56:37.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:37 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:56:37.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:56:37.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:56:37.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:56:38.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 167791 14.73 MB/sec execute 494 sec latency 187.077 ms 2024-09-13T15:56:38.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 87723 7.60 MB/sec execute 494 sec latency 71.855 ms 2024-09-13T15:56:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:38 smithi090 ceph-mon[27811]: pgmap v939: 129 pgs: 129 active+clean; 428 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 MiB/s wr, 189 op/s 2024-09-13T15:56:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:38 smithi175 ceph-mon[36128]: pgmap v939: 129 pgs: 129 active+clean; 428 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 3.0 MiB/s wr, 189 op/s 2024-09-13T15:56:39.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 167978 14.72 MB/sec execute 495 sec latency 546.213 ms 2024-09-13T15:56:39.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 87889 7.61 MB/sec execute 495 sec latency 148.689 ms 2024-09-13T15:56:40.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 168273 14.72 MB/sec execute 496 sec latency 849.321 ms 2024-09-13T15:56:40.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88039 7.61 MB/sec execute 496 sec latency 136.265 ms 2024-09-13T15:56:40.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:40 smithi090 ceph-mon[27811]: pgmap v940: 129 pgs: 129 active+clean; 434 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-13T15:56:40.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:40 smithi175 ceph-mon[36128]: pgmap v940: 129 pgs: 129 active+clean; 434 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.0 KiB/s rd, 3.2 MiB/s wr, 219 op/s 2024-09-13T15:56:41.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 168566 14.71 MB/sec execute 497 sec latency 648.582 ms 2024-09-13T15:56:41.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88143 7.61 MB/sec execute 497 sec latency 134.630 ms 2024-09-13T15:56:42.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 168604 14.68 MB/sec execute 498 sec latency 710.892 ms 2024-09-13T15:56:42.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88252 7.61 MB/sec execute 498 sec latency 189.890 ms 2024-09-13T15:56:42.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:42 smithi090 ceph-mon[27811]: pgmap v941: 129 pgs: 129 active+clean; 435 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-09-13T15:56:42.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:42 smithi175 ceph-mon[36128]: pgmap v941: 129 pgs: 129 active+clean; 435 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.3 KiB/s rd, 2.7 MiB/s wr, 174 op/s 2024-09-13T15:56:43.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 168978 14.69 MB/sec execute 499 sec latency 617.495 ms 2024-09-13T15:56:43.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88369 7.62 MB/sec execute 499 sec latency 171.580 ms 2024-09-13T15:56:44.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:44 smithi090 ceph-mon[27811]: pgmap v942: 129 pgs: 129 active+clean; 439 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-09-13T15:56:44.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 169106 14.67 MB/sec execute 500 sec latency 817.487 ms 2024-09-13T15:56:44.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88542 7.63 MB/sec execute 500 sec latency 179.809 ms 2024-09-13T15:56:44.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:44 smithi175 ceph-mon[36128]: pgmap v942: 129 pgs: 129 active+clean; 439 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.8 MiB/s wr, 196 op/s 2024-09-13T15:56:45.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 169374 14.69 MB/sec execute 501 sec latency 675.479 ms 2024-09-13T15:56:45.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88695 7.64 MB/sec execute 501 sec latency 163.355 ms 2024-09-13T15:56:45.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:45 smithi090 ceph-mon[27811]: pgmap v943: 129 pgs: 129 active+clean; 443 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-09-13T15:56:45.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:45 smithi175 ceph-mon[36128]: pgmap v943: 129 pgs: 129 active+clean; 443 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.8 MiB/s wr, 188 op/s 2024-09-13T15:56:46.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 169570 14.69 MB/sec execute 502 sec latency 689.730 ms 2024-09-13T15:56:46.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88810 7.63 MB/sec execute 502 sec latency 88.087 ms 2024-09-13T15:56:47.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 169675 14.67 MB/sec execute 503 sec latency 568.620 ms 2024-09-13T15:56:47.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 88894 7.62 MB/sec execute 503 sec latency 148.589 ms 2024-09-13T15:56:48.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:47 smithi090 ceph-mon[27811]: pgmap v944: 129 pgs: 129 active+clean; 447 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-09-13T15:56:48.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:47 smithi175 ceph-mon[36128]: pgmap v944: 129 pgs: 129 active+clean; 447 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.2 KiB/s rd, 3.8 MiB/s wr, 143 op/s 2024-09-13T15:56:48.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 169863 14.67 MB/sec execute 504 sec latency 578.361 ms 2024-09-13T15:56:48.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89013 7.62 MB/sec execute 504 sec latency 126.929 ms 2024-09-13T15:56:49.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:48 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:49.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:48 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:56:49.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 170127 14.66 MB/sec execute 505 sec latency 678.554 ms 2024-09-13T15:56:49.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89153 7.63 MB/sec execute 505 sec latency 175.958 ms 2024-09-13T15:56:50.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:49 smithi090 ceph-mon[27811]: pgmap v945: 129 pgs: 129 active+clean; 452 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 170 op/s 2024-09-13T15:56:50.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:49 smithi175 ceph-mon[36128]: pgmap v945: 129 pgs: 129 active+clean; 452 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.3 MiB/s wr, 170 op/s 2024-09-13T15:56:50.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 170236 14.64 MB/sec execute 506 sec latency 764.191 ms 2024-09-13T15:56:50.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89257 7.63 MB/sec execute 506 sec latency 351.881 ms 2024-09-13T15:56:51.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 170717 14.67 MB/sec execute 507 sec latency 798.052 ms 2024-09-13T15:56:51.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89384 7.62 MB/sec execute 507 sec latency 151.415 ms 2024-09-13T15:56:52.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:51 smithi090 ceph-mon[27811]: pgmap v946: 129 pgs: 129 active+clean; 452 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.7 MiB/s wr, 115 op/s 2024-09-13T15:56:52.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:51 smithi175 ceph-mon[36128]: pgmap v946: 129 pgs: 129 active+clean; 452 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.7 MiB/s wr, 115 op/s 2024-09-13T15:56:52.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 170735 14.65 MB/sec execute 508 sec latency 815.022 ms 2024-09-13T15:56:52.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89517 7.62 MB/sec execute 508 sec latency 122.501 ms 2024-09-13T15:56:53.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 170903 14.63 MB/sec execute 509 sec latency 819.037 ms 2024-09-13T15:56:53.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89638 7.61 MB/sec execute 509 sec latency 147.673 ms 2024-09-13T15:56:54.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:53 smithi090 ceph-mon[27811]: pgmap v947: 129 pgs: 129 active+clean; 447 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 163 op/s 2024-09-13T15:56:54.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:53 smithi175 ceph-mon[36128]: pgmap v947: 129 pgs: 129 active+clean; 447 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.5 MiB/s wr, 163 op/s 2024-09-13T15:56:54.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 171056 14.62 MB/sec execute 510 sec latency 382.341 ms 2024-09-13T15:56:54.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89793 7.60 MB/sec execute 510 sec latency 138.227 ms 2024-09-13T15:56:55.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 171144 14.59 MB/sec execute 511 sec latency 262.403 ms 2024-09-13T15:56:55.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 89947 7.60 MB/sec execute 511 sec latency 192.483 ms 2024-09-13T15:56:56.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:55 smithi090 ceph-mon[27811]: pgmap v948: 129 pgs: 129 active+clean; 446 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 147 op/s 2024-09-13T15:56:56.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:55 smithi175 ceph-mon[36128]: pgmap v948: 129 pgs: 129 active+clean; 446 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 147 op/s 2024-09-13T15:56:56.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 172015 14.65 MB/sec execute 512 sec latency 183.896 ms 2024-09-13T15:56:56.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 90120 7.60 MB/sec execute 512 sec latency 126.206 ms 2024-09-13T15:56:57.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 172690 14.68 MB/sec execute 513 sec latency 196.018 ms 2024-09-13T15:56:57.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 90288 7.59 MB/sec execute 513 sec latency 88.732 ms 2024-09-13T15:56:58.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:57 smithi090 ceph-mon[27811]: pgmap v949: 129 pgs: 129 active+clean; 445 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-09-13T15:56:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:57 smithi175 ceph-mon[36128]: pgmap v949: 129 pgs: 129 active+clean; 445 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 164 op/s 2024-09-13T15:56:58.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 173458 14.76 MB/sec execute 514 sec latency 246.592 ms 2024-09-13T15:56:58.416 INFO:tasks.workunit.client.1.smithi175.stdout: 10 90476 7.60 MB/sec execute 514 sec latency 97.083 ms 2024-09-13T15:56:59.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174225 14.81 MB/sec execute 515 sec latency 670.842 ms 2024-09-13T15:56:59.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 90655 7.61 MB/sec execute 515 sec latency 115.426 ms 2024-09-13T15:57:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:56:59 smithi090 ceph-mon[27811]: pgmap v950: 129 pgs: 129 active+clean; 434 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.7 MiB/s wr, 213 op/s 2024-09-13T15:57:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:56:59 smithi175 ceph-mon[36128]: pgmap v950: 129 pgs: 129 active+clean; 434 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 3.7 MiB/s wr, 213 op/s 2024-09-13T15:57:00.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174267 14.79 MB/sec execute 516 sec latency 542.591 ms 2024-09-13T15:57:00.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 90809 7.60 MB/sec execute 516 sec latency 180.043 ms 2024-09-13T15:57:01.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174381 14.77 MB/sec execute 517 sec latency 579.151 ms 2024-09-13T15:57:01.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 90939 7.60 MB/sec execute 517 sec latency 161.747 ms 2024-09-13T15:57:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:01 smithi090 ceph-mon[27811]: pgmap v951: 129 pgs: 129 active+clean; 433 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-09-13T15:57:02.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:01 smithi175 ceph-mon[36128]: pgmap v951: 129 pgs: 129 active+clean; 433 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 2.9 MiB/s wr, 173 op/s 2024-09-13T15:57:02.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174415 14.74 MB/sec execute 518 sec latency 592.144 ms 2024-09-13T15:57:02.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91065 7.59 MB/sec execute 518 sec latency 142.838 ms 2024-09-13T15:57:03.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174455 14.72 MB/sec execute 519 sec latency 522.097 ms 2024-09-13T15:57:03.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91203 7.59 MB/sec execute 519 sec latency 356.534 ms 2024-09-13T15:57:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:03 smithi090 ceph-mon[27811]: pgmap v952: 129 pgs: 129 active+clean; 433 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-09-13T15:57:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:03 smithi175 ceph-mon[36128]: pgmap v952: 129 pgs: 129 active+clean; 433 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.1 KiB/s rd, 3.7 MiB/s wr, 229 op/s 2024-09-13T15:57:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:04.379 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174463 14.69 MB/sec execute 520 sec latency 553.333 ms 2024-09-13T15:57:04.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91380 7.61 MB/sec execute 520 sec latency 128.717 ms 2024-09-13T15:57:05.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174570 14.67 MB/sec execute 521 sec latency 507.947 ms 2024-09-13T15:57:05.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91530 7.60 MB/sec execute 521 sec latency 201.165 ms 2024-09-13T15:57:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:05 smithi090 ceph-mon[27811]: pgmap v953: 129 pgs: 129 active+clean; 434 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.9 MiB/s wr, 190 op/s 2024-09-13T15:57:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:05 smithi175 ceph-mon[36128]: pgmap v953: 129 pgs: 129 active+clean; 434 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 2.9 MiB/s wr, 190 op/s 2024-09-13T15:57:06.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174607 14.65 MB/sec execute 522 sec latency 553.252 ms 2024-09-13T15:57:06.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91654 7.60 MB/sec execute 522 sec latency 178.822 ms 2024-09-13T15:57:07.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174612 14.62 MB/sec execute 523 sec latency 547.667 ms 2024-09-13T15:57:07.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91743 7.61 MB/sec execute 523 sec latency 165.277 ms 2024-09-13T15:57:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:07 smithi090 ceph-mon[27811]: pgmap v954: 129 pgs: 129 active+clean; 437 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.1 MiB/s wr, 193 op/s 2024-09-13T15:57:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:07 smithi175 ceph-mon[36128]: pgmap v954: 129 pgs: 129 active+clean; 437 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.1 MiB/s wr, 193 op/s 2024-09-13T15:57:08.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174638 14.59 MB/sec execute 524 sec latency 560.755 ms 2024-09-13T15:57:08.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 91886 7.62 MB/sec execute 524 sec latency 168.336 ms 2024-09-13T15:57:09.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174691 14.57 MB/sec execute 525 sec latency 596.720 ms 2024-09-13T15:57:09.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92057 7.62 MB/sec execute 525 sec latency 118.788 ms 2024-09-13T15:57:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:09 smithi090 ceph-mon[27811]: pgmap v955: 129 pgs: 129 active+clean; 442 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.3 MiB/s wr, 160 op/s 2024-09-13T15:57:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:09 smithi175 ceph-mon[36128]: pgmap v955: 129 pgs: 129 active+clean; 442 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.3 KiB/s rd, 3.3 MiB/s wr, 160 op/s 2024-09-13T15:57:10.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174723 14.54 MB/sec execute 526 sec latency 645.941 ms 2024-09-13T15:57:10.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92209 7.63 MB/sec execute 526 sec latency 205.545 ms 2024-09-13T15:57:11.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174827 14.52 MB/sec execute 527 sec latency 695.920 ms 2024-09-13T15:57:11.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92331 7.63 MB/sec execute 527 sec latency 143.513 ms 2024-09-13T15:57:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:11 smithi175 ceph-mon[36128]: pgmap v956: 129 pgs: 129 active+clean; 446 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-09-13T15:57:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:11 smithi090 ceph-mon[27811]: pgmap v956: 129 pgs: 129 active+clean; 446 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 3.3 KiB/s rd, 3.2 MiB/s wr, 138 op/s 2024-09-13T15:57:12.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 174909 14.50 MB/sec execute 528 sec latency 507.976 ms 2024-09-13T15:57:12.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92402 7.62 MB/sec execute 528 sec latency 139.892 ms 2024-09-13T15:57:13.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 175338 14.53 MB/sec execute 529 sec latency 696.351 ms 2024-09-13T15:57:13.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92512 7.62 MB/sec execute 529 sec latency 115.636 ms 2024-09-13T15:57:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:13 smithi090 ceph-mon[27811]: pgmap v957: 129 pgs: 129 active+clean; 451 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 147 op/s 2024-09-13T15:57:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:13 smithi175 ceph-mon[36128]: pgmap v957: 129 pgs: 129 active+clean; 451 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.0 MiB/s wr, 147 op/s 2024-09-13T15:57:14.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 176016 14.54 MB/sec execute 530 sec latency 635.278 ms 2024-09-13T15:57:14.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92659 7.62 MB/sec execute 530 sec latency 125.343 ms 2024-09-13T15:57:15.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 176714 14.61 MB/sec execute 531 sec latency 288.344 ms 2024-09-13T15:57:15.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92815 7.63 MB/sec execute 531 sec latency 165.670 ms 2024-09-13T15:57:16.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:15 smithi090 ceph-mon[27811]: pgmap v958: 129 pgs: 129 active+clean; 455 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.6 MiB/s wr, 116 op/s 2024-09-13T15:57:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:15 smithi175 ceph-mon[36128]: pgmap v958: 129 pgs: 129 active+clean; 455 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.6 MiB/s wr, 116 op/s 2024-09-13T15:57:16.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 177810 14.70 MB/sec execute 532 sec latency 340.647 ms 2024-09-13T15:57:16.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 92955 7.62 MB/sec execute 532 sec latency 136.461 ms 2024-09-13T15:57:17.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178010 14.69 MB/sec execute 533 sec latency 226.669 ms 2024-09-13T15:57:17.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93092 7.62 MB/sec execute 533 sec latency 117.652 ms 2024-09-13T15:57:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:17 smithi090 ceph-mon[27811]: pgmap v959: 129 pgs: 129 active+clean; 450 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.9 MiB/s wr, 115 op/s 2024-09-13T15:57:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:17 smithi175 ceph-mon[36128]: pgmap v959: 129 pgs: 129 active+clean; 450 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.9 MiB/s wr, 115 op/s 2024-09-13T15:57:18.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178175 14.68 MB/sec execute 534 sec latency 244.480 ms 2024-09-13T15:57:18.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93228 7.61 MB/sec execute 534 sec latency 134.002 ms 2024-09-13T15:57:19.253 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:19.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178260 14.65 MB/sec execute 535 sec latency 516.424 ms 2024-09-13T15:57:19.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93362 7.60 MB/sec execute 535 sec latency 144.783 ms 2024-09-13T15:57:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:19 smithi090 ceph-mon[27811]: pgmap v960: 129 pgs: 129 active+clean; 446 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-09-13T15:57:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:19 smithi175 ceph-mon[36128]: pgmap v960: 129 pgs: 129 active+clean; 446 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.1 MiB/s wr, 154 op/s 2024-09-13T15:57:20.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178277 14.63 MB/sec execute 536 sec latency 748.889 ms 2024-09-13T15:57:20.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93533 7.61 MB/sec execute 536 sec latency 124.816 ms 2024-09-13T15:57:21.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178391 14.61 MB/sec execute 537 sec latency 844.943 ms 2024-09-13T15:57:21.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93672 7.60 MB/sec execute 537 sec latency 187.273 ms 2024-09-13T15:57:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:21 smithi090 ceph-mon[27811]: pgmap v961: 129 pgs: 129 active+clean; 448 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 167 op/s 2024-09-13T15:57:22.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:21 smithi175 ceph-mon[36128]: pgmap v961: 129 pgs: 129 active+clean; 448 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 167 op/s 2024-09-13T15:57:22.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178744 14.64 MB/sec execute 538 sec latency 843.712 ms 2024-09-13T15:57:22.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93817 7.59 MB/sec execute 538 sec latency 111.696 ms 2024-09-13T15:57:23.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 178883 14.61 MB/sec execute 539 sec latency 715.099 ms 2024-09-13T15:57:23.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 93990 7.60 MB/sec execute 539 sec latency 156.742 ms 2024-09-13T15:57:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:23 smithi090 ceph-mon[27811]: pgmap v962: 129 pgs: 129 active+clean; 436 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-09-13T15:57:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:23 smithi175 ceph-mon[36128]: pgmap v962: 129 pgs: 129 active+clean; 436 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.9 KiB/s rd, 4.0 MiB/s wr, 184 op/s 2024-09-13T15:57:24.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 179108 14.59 MB/sec execute 540 sec latency 766.637 ms 2024-09-13T15:57:24.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 94137 7.60 MB/sec execute 540 sec latency 197.930 ms 2024-09-13T15:57:25.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 179182 14.57 MB/sec execute 541 sec latency 706.616 ms 2024-09-13T15:57:25.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 94281 7.60 MB/sec execute 541 sec latency 173.421 ms 2024-09-13T15:57:26.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:25 smithi090 ceph-mon[27811]: pgmap v963: 129 pgs: 129 active+clean; 430 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-09-13T15:57:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:25 smithi175 ceph-mon[36128]: pgmap v963: 129 pgs: 129 active+clean; 430 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.4 MiB/s wr, 208 op/s 2024-09-13T15:57:26.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 179550 14.57 MB/sec execute 542 sec latency 756.581 ms 2024-09-13T15:57:26.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 94417 7.60 MB/sec execute 542 sec latency 225.450 ms 2024-09-13T15:57:27.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 179622 14.55 MB/sec execute 543 sec latency 645.379 ms 2024-09-13T15:57:27.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 94563 7.59 MB/sec execute 543 sec latency 149.994 ms 2024-09-13T15:57:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:27 smithi090 ceph-mon[27811]: pgmap v964: 129 pgs: 129 active+clean; 430 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.2 MiB/s wr, 195 op/s 2024-09-13T15:57:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:27 smithi175 ceph-mon[36128]: pgmap v964: 129 pgs: 129 active+clean; 430 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.2 MiB/s wr, 195 op/s 2024-09-13T15:57:28.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 179982 14.58 MB/sec execute 544 sec latency 583.862 ms 2024-09-13T15:57:28.417 INFO:tasks.workunit.client.1.smithi175.stdout: 10 94706 7.59 MB/sec execute 544 sec latency 159.099 ms 2024-09-13T15:57:29.380 INFO:tasks.workunit.client.0.smithi090.stdout: 10 180149 14.58 MB/sec execute 545 sec latency 818.749 ms 2024-09-13T15:57:29.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 94900 7.61 MB/sec execute 545 sec latency 101.892 ms 2024-09-13T15:57:30.015 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:29 smithi175 ceph-mon[36128]: pgmap v965: 129 pgs: 129 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.4 MiB/s wr, 231 op/s 2024-09-13T15:57:30.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:29 smithi090 ceph-mon[27811]: pgmap v965: 129 pgs: 129 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.4 MiB/s wr, 231 op/s 2024-09-13T15:57:30.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 180376 14.57 MB/sec execute 546 sec latency 703.888 ms 2024-09-13T15:57:30.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95074 7.61 MB/sec execute 546 sec latency 162.357 ms 2024-09-13T15:57:31.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 180478 14.56 MB/sec execute 547 sec latency 762.238 ms 2024-09-13T15:57:31.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95198 7.61 MB/sec execute 547 sec latency 209.834 ms 2024-09-13T15:57:32.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:31 smithi090 ceph-mon[27811]: pgmap v966: 129 pgs: 129 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.7 MiB/s wr, 183 op/s 2024-09-13T15:57:32.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:31 smithi175 ceph-mon[36128]: pgmap v966: 129 pgs: 129 active+clean; 432 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 6.0 KiB/s rd, 2.7 MiB/s wr, 183 op/s 2024-09-13T15:57:32.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 180558 14.54 MB/sec execute 548 sec latency 674.719 ms 2024-09-13T15:57:32.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95317 7.61 MB/sec execute 548 sec latency 124.963 ms 2024-09-13T15:57:33.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 181226 14.58 MB/sec execute 549 sec latency 773.550 ms 2024-09-13T15:57:33.455 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95456 7.62 MB/sec execute 549 sec latency 147.098 ms 2024-09-13T15:57:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:33 smithi090 ceph-mon[27811]: pgmap v967: 129 pgs: 129 active+clean; 442 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 190 op/s 2024-09-13T15:57:34.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:33 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:33 smithi175 ceph-mon[36128]: pgmap v967: 129 pgs: 129 active+clean; 442 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.6 MiB/s wr, 190 op/s 2024-09-13T15:57:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:33 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:34.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 181357 14.57 MB/sec execute 550 sec latency 717.690 ms 2024-09-13T15:57:34.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95624 7.63 MB/sec execute 550 sec latency 201.484 ms 2024-09-13T15:57:35.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 181381 14.54 MB/sec execute 551 sec latency 882.310 ms 2024-09-13T15:57:35.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95777 7.64 MB/sec execute 551 sec latency 277.701 ms 2024-09-13T15:57:36.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:35 smithi090 ceph-mon[27811]: pgmap v968: 129 pgs: 129 active+clean; 445 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 172 op/s 2024-09-13T15:57:36.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:35 smithi175 ceph-mon[36128]: pgmap v968: 129 pgs: 129 active+clean; 445 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.4 MiB/s wr, 172 op/s 2024-09-13T15:57:36.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 181557 14.53 MB/sec execute 552 sec latency 485.448 ms 2024-09-13T15:57:36.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95904 7.63 MB/sec execute 552 sec latency 132.328 ms 2024-09-13T15:57:37.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:57:37.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:37.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:36 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:37.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:57:37.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:37.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:37 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:37.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 181723 14.52 MB/sec execute 553 sec latency 302.048 ms 2024-09-13T15:57:37.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 95984 7.63 MB/sec execute 553 sec latency 130.124 ms 2024-09-13T15:57:38.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 182579 14.56 MB/sec execute 554 sec latency 117.897 ms 2024-09-13T15:57:38.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96111 7.62 MB/sec execute 554 sec latency 96.759 ms 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: pgmap v969: 129 pgs: 129 active+clean; 448 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.5 MiB/s wr, 141 op/s 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:57:38.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: pgmap v969: 129 pgs: 129 active+clean; 448 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.5 MiB/s wr, 141 op/s 2024-09-13T15:57:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:38.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:57:38.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:57:38.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:57:39.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 183735 14.66 MB/sec execute 555 sec latency 115.061 ms 2024-09-13T15:57:39.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96263 7.63 MB/sec execute 555 sec latency 109.530 ms 2024-09-13T15:57:40.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:40 smithi175 ceph-mon[36128]: pgmap v970: 129 pgs: 129 active+clean; 447 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.0 MiB/s wr, 143 op/s 2024-09-13T15:57:40.327 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:40 smithi090 ceph-mon[27811]: pgmap v970: 129 pgs: 129 active+clean; 447 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.4 KiB/s rd, 4.0 MiB/s wr, 143 op/s 2024-09-13T15:57:40.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 184901 14.75 MB/sec execute 556 sec latency 222.934 ms 2024-09-13T15:57:40.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96427 7.63 MB/sec execute 556 sec latency 116.296 ms 2024-09-13T15:57:41.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185082 14.74 MB/sec execute 557 sec latency 480.870 ms 2024-09-13T15:57:41.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96549 7.63 MB/sec execute 557 sec latency 128.535 ms 2024-09-13T15:57:42.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185094 14.72 MB/sec execute 558 sec latency 588.546 ms 2024-09-13T15:57:42.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96662 7.62 MB/sec execute 558 sec latency 165.815 ms 2024-09-13T15:57:42.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:42 smithi090 ceph-mon[27811]: pgmap v971: 129 pgs: 129 active+clean; 450 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 121 op/s 2024-09-13T15:57:42.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:42 smithi175 ceph-mon[36128]: pgmap v971: 129 pgs: 129 active+clean; 450 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 3.7 KiB/s rd, 3.8 MiB/s wr, 121 op/s 2024-09-13T15:57:43.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185189 14.70 MB/sec execute 559 sec latency 638.508 ms 2024-09-13T15:57:43.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96786 7.61 MB/sec execute 559 sec latency 167.180 ms 2024-09-13T15:57:44.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185232 14.68 MB/sec execute 560 sec latency 672.949 ms 2024-09-13T15:57:44.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 96933 7.61 MB/sec execute 560 sec latency 131.212 ms 2024-09-13T15:57:44.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:44 smithi090 ceph-mon[27811]: pgmap v972: 129 pgs: 129 active+clean; 446 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.5 MiB/s wr, 144 op/s 2024-09-13T15:57:44.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:44 smithi175 ceph-mon[36128]: pgmap v972: 129 pgs: 129 active+clean; 446 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.5 MiB/s wr, 144 op/s 2024-09-13T15:57:45.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185242 14.65 MB/sec execute 561 sec latency 705.695 ms 2024-09-13T15:57:45.418 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97079 7.61 MB/sec execute 561 sec latency 174.931 ms 2024-09-13T15:57:46.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185252 14.63 MB/sec execute 562 sec latency 564.451 ms 2024-09-13T15:57:46.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97208 7.60 MB/sec execute 562 sec latency 160.297 ms 2024-09-13T15:57:46.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:46 smithi090 ceph-mon[27811]: pgmap v973: 129 pgs: 129 active+clean; 444 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 158 op/s 2024-09-13T15:57:46.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:46 smithi175 ceph-mon[36128]: pgmap v973: 129 pgs: 129 active+clean; 444 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 158 op/s 2024-09-13T15:57:47.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185303 14.60 MB/sec execute 563 sec latency 587.197 ms 2024-09-13T15:57:47.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97343 7.59 MB/sec execute 563 sec latency 187.674 ms 2024-09-13T15:57:48.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185345 14.58 MB/sec execute 564 sec latency 511.270 ms 2024-09-13T15:57:48.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97524 7.60 MB/sec execute 564 sec latency 124.867 ms 2024-09-13T15:57:48.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:48 smithi090 ceph-mon[27811]: pgmap v974: 129 pgs: 129 active+clean; 439 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-09-13T15:57:48.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:48 smithi175 ceph-mon[36128]: pgmap v974: 129 pgs: 129 active+clean; 439 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 182 op/s 2024-09-13T15:57:49.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185368 14.55 MB/sec execute 565 sec latency 671.138 ms 2024-09-13T15:57:49.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97664 7.60 MB/sec execute 565 sec latency 131.577 ms 2024-09-13T15:57:49.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:49.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:57:50.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185486 14.54 MB/sec execute 566 sec latency 504.311 ms 2024-09-13T15:57:50.416 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:50 smithi175 ceph-mon[36128]: pgmap v975: 129 pgs: 129 active+clean; 432 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.6 MiB/s wr, 201 op/s 2024-09-13T15:57:50.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97828 7.60 MB/sec execute 566 sec latency 151.140 ms 2024-09-13T15:57:50.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:50 smithi090 ceph-mon[27811]: pgmap v975: 129 pgs: 129 active+clean; 432 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.6 MiB/s wr, 201 op/s 2024-09-13T15:57:51.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 185956 14.56 MB/sec execute 567 sec latency 681.978 ms 2024-09-13T15:57:51.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 97981 7.60 MB/sec execute 567 sec latency 165.996 ms 2024-09-13T15:57:52.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 186174 14.55 MB/sec execute 568 sec latency 522.644 ms 2024-09-13T15:57:52.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98110 7.60 MB/sec execute 568 sec latency 168.508 ms 2024-09-13T15:57:52.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:52 smithi090 ceph-mon[27811]: pgmap v976: 129 pgs: 129 active+clean; 431 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-09-13T15:57:52.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:52 smithi175 ceph-mon[36128]: pgmap v976: 129 pgs: 129 active+clean; 431 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 7.7 KiB/s rd, 2.9 MiB/s wr, 191 op/s 2024-09-13T15:57:53.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 186268 14.53 MB/sec execute 569 sec latency 708.162 ms 2024-09-13T15:57:53.419 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98252 7.59 MB/sec execute 569 sec latency 186.358 ms 2024-09-13T15:57:54.381 INFO:tasks.workunit.client.0.smithi090.stdout: 10 186640 14.52 MB/sec execute 570 sec latency 806.747 ms 2024-09-13T15:57:54.420 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98447 7.61 MB/sec execute 570 sec latency 126.973 ms 2024-09-13T15:57:54.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:54 smithi090 ceph-mon[27811]: pgmap v977: 129 pgs: 129 active+clean; 437 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-09-13T15:57:54.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:54 smithi175 ceph-mon[36128]: pgmap v977: 129 pgs: 129 active+clean; 437 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 3.4 MiB/s wr, 206 op/s 2024-09-13T15:57:55.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 186794 14.51 MB/sec execute 571 sec latency 528.283 ms 2024-09-13T15:57:55.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98595 7.61 MB/sec execute 571 sec latency 142.552 ms 2024-09-13T15:57:56.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 187183 14.56 MB/sec execute 572 sec latency 733.241 ms 2024-09-13T15:57:56.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98714 7.61 MB/sec execute 572 sec latency 209.635 ms 2024-09-13T15:57:56.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:56 smithi090 ceph-mon[27811]: pgmap v978: 129 pgs: 129 active+clean; 435 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-09-13T15:57:56.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:56 smithi175 ceph-mon[36128]: pgmap v978: 129 pgs: 129 active+clean; 435 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 215 op/s 2024-09-13T15:57:57.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 187626 14.57 MB/sec execute 573 sec latency 440.051 ms 2024-09-13T15:57:57.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98839 7.61 MB/sec execute 573 sec latency 171.507 ms 2024-09-13T15:57:57.461 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:57 smithi090 ceph-mon[27811]: pgmap v979: 129 pgs: 129 active+clean; 439 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-09-13T15:57:57.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:57 smithi175 ceph-mon[36128]: pgmap v979: 129 pgs: 129 active+clean; 439 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 176 op/s 2024-09-13T15:57:58.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 188603 14.64 MB/sec execute 574 sec latency 143.112 ms 2024-09-13T15:57:58.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 98978 7.61 MB/sec execute 574 sec latency 83.462 ms 2024-09-13T15:57:59.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 188748 14.62 MB/sec execute 575 sec latency 400.573 ms 2024-09-13T15:57:59.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99186 7.63 MB/sec execute 575 sec latency 115.079 ms 2024-09-13T15:58:00.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:57:59 smithi090 ceph-mon[27811]: pgmap v980: 129 pgs: 129 active+clean; 441 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-09-13T15:58:00.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:57:59 smithi175 ceph-mon[36128]: pgmap v980: 129 pgs: 129 active+clean; 441 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 7.4 KiB/s rd, 3.2 MiB/s wr, 159 op/s 2024-09-13T15:58:00.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 188921 14.61 MB/sec execute 576 sec latency 117.808 ms 2024-09-13T15:58:00.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99382 7.64 MB/sec execute 576 sec latency 123.022 ms 2024-09-13T15:58:01.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 190172 14.68 MB/sec execute 577 sec latency 120.062 ms 2024-09-13T15:58:01.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99489 7.64 MB/sec execute 577 sec latency 96.406 ms 2024-09-13T15:58:02.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:01 smithi090 ceph-mon[27811]: pgmap v981: 129 pgs: 129 active+clean; 443 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.3 MiB/s wr, 149 op/s 2024-09-13T15:58:02.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:01 smithi175 ceph-mon[36128]: pgmap v981: 129 pgs: 129 active+clean; 443 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 3.4 KiB/s rd, 3.3 MiB/s wr, 149 op/s 2024-09-13T15:58:02.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 190977 14.75 MB/sec execute 578 sec latency 154.415 ms 2024-09-13T15:58:02.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99590 7.63 MB/sec execute 578 sec latency 89.574 ms 2024-09-13T15:58:03.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 191179 14.75 MB/sec execute 579 sec latency 498.677 ms 2024-09-13T15:58:03.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99692 7.63 MB/sec execute 579 sec latency 177.677 ms 2024-09-13T15:58:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:03 smithi090 ceph-mon[27811]: pgmap v982: 129 pgs: 129 active+clean; 448 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 162 op/s 2024-09-13T15:58:04.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:03 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:03 smithi175 ceph-mon[36128]: pgmap v982: 129 pgs: 129 active+clean; 448 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 162 op/s 2024-09-13T15:58:04.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:03 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:04.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 191316 14.74 MB/sec execute 580 sec latency 601.731 ms 2024-09-13T15:58:04.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99820 7.64 MB/sec execute 580 sec latency 148.051 ms 2024-09-13T15:58:05.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 191986 14.78 MB/sec execute 581 sec latency 575.269 ms 2024-09-13T15:58:05.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 99977 7.63 MB/sec execute 581 sec latency 131.810 ms 2024-09-13T15:58:06.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:05 smithi090 ceph-mon[27811]: pgmap v983: 129 pgs: 129 active+clean; 452 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.1 MiB/s wr, 160 op/s 2024-09-13T15:58:06.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:05 smithi175 ceph-mon[36128]: pgmap v983: 129 pgs: 129 active+clean; 452 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.1 MiB/s wr, 160 op/s 2024-09-13T15:58:06.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192007 14.75 MB/sec execute 582 sec latency 775.187 ms 2024-09-13T15:58:06.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100106 7.63 MB/sec execute 582 sec latency 202.882 ms 2024-09-13T15:58:07.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192111 14.74 MB/sec execute 583 sec latency 591.650 ms 2024-09-13T15:58:07.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100208 7.62 MB/sec execute 583 sec latency 185.249 ms 2024-09-13T15:58:08.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:07 smithi090 ceph-mon[27811]: pgmap v984: 129 pgs: 129 active+clean; 445 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 138 op/s 2024-09-13T15:58:08.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:07 smithi175 ceph-mon[36128]: pgmap v984: 129 pgs: 129 active+clean; 445 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.2 MiB/s wr, 138 op/s 2024-09-13T15:58:08.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192144 14.72 MB/sec execute 584 sec latency 749.273 ms 2024-09-13T15:58:08.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100313 7.61 MB/sec execute 584 sec latency 151.061 ms 2024-09-13T15:58:09.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192181 14.69 MB/sec execute 585 sec latency 531.687 ms 2024-09-13T15:58:09.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100438 7.60 MB/sec execute 585 sec latency 128.424 ms 2024-09-13T15:58:10.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:09 smithi090 ceph-mon[27811]: pgmap v985: 129 pgs: 129 active+clean; 445 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.3 MiB/s wr, 179 op/s 2024-09-13T15:58:10.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:09 smithi175 ceph-mon[36128]: pgmap v985: 129 pgs: 129 active+clean; 445 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 8.2 KiB/s rd, 4.3 MiB/s wr, 179 op/s 2024-09-13T15:58:10.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192186 14.67 MB/sec execute 586 sec latency 583.488 ms 2024-09-13T15:58:10.425 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100557 7.60 MB/sec execute 586 sec latency 196.798 ms 2024-09-13T15:58:11.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192287 14.65 MB/sec execute 587 sec latency 537.945 ms 2024-09-13T15:58:11.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100701 7.60 MB/sec execute 587 sec latency 138.388 ms 2024-09-13T15:58:11.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:11 smithi175 ceph-mon[36128]: pgmap v986: 129 pgs: 129 active+clean; 444 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.6 MiB/s wr, 147 op/s 2024-09-13T15:58:12.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:11 smithi090 ceph-mon[27811]: pgmap v986: 129 pgs: 129 active+clean; 444 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.6 MiB/s wr, 147 op/s 2024-09-13T15:58:12.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192330 14.63 MB/sec execute 588 sec latency 626.660 ms 2024-09-13T15:58:12.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100797 7.59 MB/sec execute 588 sec latency 241.680 ms 2024-09-13T15:58:13.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192335 14.60 MB/sec execute 589 sec latency 508.369 ms 2024-09-13T15:58:13.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 100937 7.59 MB/sec execute 589 sec latency 165.228 ms 2024-09-13T15:58:14.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:13 smithi090 ceph-mon[27811]: pgmap v987: 129 pgs: 129 active+clean; 432 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.9 MiB/s wr, 173 op/s 2024-09-13T15:58:14.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:13 smithi175 ceph-mon[36128]: pgmap v987: 129 pgs: 129 active+clean; 432 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 9.7 KiB/s rd, 3.9 MiB/s wr, 173 op/s 2024-09-13T15:58:14.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192354 14.58 MB/sec execute 590 sec latency 602.991 ms 2024-09-13T15:58:14.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 101114 7.60 MB/sec execute 590 sec latency 193.469 ms 2024-09-13T15:58:15.383 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192372 14.56 MB/sec execute 591 sec latency 604.498 ms 2024-09-13T15:58:15.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 101279 7.59 MB/sec execute 591 sec latency 99.693 ms 2024-09-13T15:58:16.223 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:15 smithi090 ceph-mon[27811]: pgmap v988: 129 pgs: 129 active+clean; 433 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.1 MiB/s wr, 198 op/s 2024-09-13T15:58:16.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:15 smithi175 ceph-mon[36128]: pgmap v988: 129 pgs: 129 active+clean; 433 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.1 MiB/s wr, 198 op/s 2024-09-13T15:58:16.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192425 14.53 MB/sec execute 592 sec latency 583.926 ms 2024-09-13T15:58:16.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 101421 7.60 MB/sec execute 592 sec latency 250.318 ms 2024-09-13T15:58:17.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192518 14.52 MB/sec execute 593 sec latency 500.045 ms 2024-09-13T15:58:17.421 INFO:tasks.workunit.client.1.smithi175.stdout: 10 101568 7.60 MB/sec execute 593 sec latency 131.493 ms 2024-09-13T15:58:18.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:17 smithi090 ceph-mon[27811]: pgmap v989: 129 pgs: 129 active+clean; 432 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-09-13T15:58:18.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:17 smithi175 ceph-mon[36128]: pgmap v989: 129 pgs: 129 active+clean; 432 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 2.7 MiB/s wr, 175 op/s 2024-09-13T15:58:18.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 192627 14.50 MB/sec execute 594 sec latency 549.859 ms 2024-09-13T15:58:18.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 101720 7.59 MB/sec execute 594 sec latency 165.591 ms 2024-09-13T15:58:19.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:18 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:19.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:18 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:19.382 INFO:tasks.workunit.client.0.smithi090.stdout: 10 193358 14.53 MB/sec execute 595 sec latency 529.858 ms 2024-09-13T15:58:19.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 101873 7.60 MB/sec execute 595 sec latency 166.091 ms 2024-09-13T15:58:20.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:19 smithi090 ceph-mon[27811]: pgmap v990: 129 pgs: 129 active+clean; 436 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-09-13T15:58:20.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:19 smithi175 ceph-mon[36128]: pgmap v990: 129 pgs: 129 active+clean; 436 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 8.3 KiB/s rd, 2.8 MiB/s wr, 192 op/s 2024-09-13T15:58:20.383 INFO:tasks.workunit.client.0.smithi090.stdout: 10 194322 14.61 MB/sec execute 596 sec latency 182.842 ms 2024-09-13T15:58:20.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 102090 7.60 MB/sec execute 596 sec latency 70.874 ms 2024-09-13T15:58:21.383 INFO:tasks.workunit.client.0.smithi090.stdout: 10 195522 14.70 MB/sec execute 597 sec latency 189.486 ms 2024-09-13T15:58:21.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 102265 7.61 MB/sec execute 597 sec latency 83.237 ms 2024-09-13T15:58:22.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:21 smithi090 ceph-mon[27811]: pgmap v991: 129 pgs: 129 active+clean; 433 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.4 MiB/s wr, 157 op/s 2024-09-13T15:58:22.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:21 smithi175 ceph-mon[36128]: pgmap v991: 129 pgs: 129 active+clean; 433 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.4 MiB/s wr, 157 op/s 2024-09-13T15:58:22.383 INFO:tasks.workunit.client.0.smithi090.stdout: 10 195835 14.70 MB/sec execute 598 sec latency 119.909 ms 2024-09-13T15:58:22.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 102403 7.62 MB/sec execute 598 sec latency 81.342 ms 2024-09-13T15:58:23.383 INFO:tasks.workunit.client.0.smithi090.stdout: 10 195963 14.68 MB/sec execute 599 sec latency 121.948 ms 2024-09-13T15:58:23.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 102543 7.61 MB/sec execute 599 sec latency 82.040 ms 2024-09-13T15:58:24.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:23 smithi090 ceph-mon[27811]: pgmap v992: 129 pgs: 129 active+clean; 439 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.6 MiB/s wr, 180 op/s 2024-09-13T15:58:24.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:23 smithi175 ceph-mon[36128]: pgmap v992: 129 pgs: 129 active+clean; 439 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.6 MiB/s wr, 180 op/s 2024-09-13T15:58:24.383 INFO:tasks.workunit.client.0.smithi090.stdout: 10 cleanup 600 sec 2024-09-13T15:58:24.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 cleanup 600 sec 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: 0 cleanup 601 sec 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: Operation Count AvgLat MaxLat 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: ---------------------------------------- 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: NTCreateX 283542 0.122 242.458 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: Close 208215 0.001 0.593 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: Rename 12029 0.798 213.272 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: Unlink 57179 1.485 366.904 2024-09-13T15:58:24.883 INFO:tasks.workunit.client.0.smithi090.stdout: Qpathinfo 257243 0.056 210.076 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: Qfileinfo 44858 0.001 0.019 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: Qfsinfo 47080 0.002 0.052 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: Sfileinfo 23182 3.360 311.868 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: Find 99320 0.027 124.357 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: WriteX 140079 0.791 311.879 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: ReadX 444487 0.042 268.094 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: LockX 920 0.003 0.011 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: UnlockX 920 0.002 0.015 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: Flush 19844 284.529 1242.658 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout: 2024-09-13T15:58:24.884 INFO:tasks.workunit.client.0.smithi090.stdout:Throughput 14.6824 MB/sec 10 clients 10 procs max_latency=1242.661 ms 2024-09-13T15:58:24.885 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T15:58:24.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-13T15:58:25.422 INFO:tasks.workunit.client.1.smithi175.stdout: 10 cleanup 601 sec 2024-09-13T15:58:25.727 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-09-13T15:58:25.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-13T15:58:26.065 INFO:tasks.workunit.client.1.smithi175.stdout: 0 cleanup 602 sec 2024-09-13T15:58:26.065 INFO:tasks.workunit.client.1.smithi175.stdout: 2024-09-13T15:58:26.065 INFO:tasks.workunit.client.1.smithi175.stdout: Operation Count AvgLat MaxLat 2024-09-13T15:58:26.065 INFO:tasks.workunit.client.1.smithi175.stdout: ---------------------------------------- 2024-09-13T15:58:26.065 INFO:tasks.workunit.client.1.smithi175.stdout: NTCreateX 147364 9.330 511.518 2024-09-13T15:58:26.065 INFO:tasks.workunit.client.1.smithi175.stdout: Close 108177 0.126 47.996 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Rename 6228 23.158 375.687 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Unlink 29795 8.273 503.130 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Qpathinfo 133475 5.252 391.511 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Qfileinfo 23238 0.227 8.209 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Qfsinfo 24426 0.607 100.728 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Sfileinfo 12007 1.745 96.262 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Find 51398 65.010 668.488 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: WriteX 72623 0.446 238.688 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: ReadX 230569 0.006 2.615 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: LockX 480 3.713 77.001 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: UnlockX 480 2.830 50.853 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: Flush 10301 9.564 480.571 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout: 2024-09-13T15:58:26.066 INFO:tasks.workunit.client.1.smithi175.stdout:Throughput 7.61388 MB/sec 10 clients 10 procs max_latency=668.492 ms 2024-09-13T15:58:26.067 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-13T15:58:26.067 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-09-13T15:58:26.240 DEBUG:teuthology.parallel:result is None 2024-09-13T15:58:26.250 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:25 smithi090 ceph-mon[27811]: pgmap v993: 129 pgs: 129 active+clean; 443 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 175 op/s 2024-09-13T15:58:26.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:25 smithi175 ceph-mon[36128]: pgmap v993: 129 pgs: 129 active+clean; 443 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.6 MiB/s wr, 175 op/s 2024-09-13T15:58:27.131 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.1... 2024-09-13T15:58:27.132 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-09-13T15:58:27.461 DEBUG:teuthology.parallel:result is None 2024-09-13T15:58:27.461 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:58:27.524 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:58:27.524 DEBUG:teuthology.orchestra.run.smithi090:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-13T15:58:27.593 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-13T15:58:27.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-09-13T15:58:27.628 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-09-13T15:58:27.628 INFO:teuthology.run_tasks:Running task pexec... 2024-09-13T15:58:27.641 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-13T15:58:27.641 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi090.front.sepia.ceph.com 2024-09-13T15:58:27.642 DEBUG:teuthology.orchestra.run.smithi090:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T15:58:27.642 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi175.front.sepia.ceph.com 2024-09-13T15:58:27.642 DEBUG:teuthology.orchestra.run.smithi175:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-13T15:58:28.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:28 smithi090 ceph-mon[27811]: pgmap v994: 129 pgs: 129 active+clean; 435 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.7 MiB/s wr, 139 op/s 2024-09-13T15:58:28.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:28 smithi175 ceph-mon[36128]: pgmap v994: 129 pgs: 129 active+clean; 435 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 3.6 KiB/s rd, 3.7 MiB/s wr, 139 op/s 2024-09-13T15:58:30.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:30 smithi175 ceph-mon[36128]: pgmap v995: 129 pgs: 129 active+clean; 337 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 188 op/s 2024-09-13T15:58:30.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:30 smithi090 ceph-mon[27811]: pgmap v995: 129 pgs: 129 active+clean; 337 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 12 KiB/s rd, 4.1 MiB/s wr, 188 op/s 2024-09-13T15:58:32.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:32 smithi175 ceph-mon[36128]: pgmap v996: 129 pgs: 129 active+clean; 314 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 MiB/s wr, 166 op/s 2024-09-13T15:58:32.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:32 smithi090 ceph-mon[27811]: pgmap v996: 129 pgs: 129 active+clean; 314 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 3.6 MiB/s wr, 166 op/s 2024-09-13T15:58:34.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:34 smithi175 ceph-mon[36128]: pgmap v997: 129 pgs: 129 active+clean; 232 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 3.3 MiB/s wr, 180 op/s 2024-09-13T15:58:34.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:34 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:34 smithi090 ceph-mon[27811]: pgmap v997: 129 pgs: 129 active+clean; 232 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 11 KiB/s rd, 3.3 MiB/s wr, 180 op/s 2024-09-13T15:58:34.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:34 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:35.221 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:58:35.222 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:58:35.222 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-13T15:58:35.222 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:Removing: 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:Removing unused dependencies: 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:Remove 2 Packages 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:Freed space: 5.5 M 2024-09-13T15:58:35.223 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:58:35.260 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:58:35.261 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:58:35.538 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:58:35.539 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:58:35.777 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:58:35.803 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout:Removing: 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout:Removing unused dependencies: 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:58:35.804 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:58:35.805 INFO:teuthology.orchestra.run.smithi090.stdout:Remove 2 Packages 2024-09-13T15:58:35.805 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:35.805 INFO:teuthology.orchestra.run.smithi090.stdout:Freed space: 5.5 M 2024-09-13T15:58:35.805 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:58:35.882 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:58:35.935 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:58:35.936 INFO:teuthology.orchestra.run.smithi175.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T15:58:36.006 INFO:teuthology.orchestra.run.smithi175.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T15:58:36.223 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:58:36.223 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:58:36.268 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:36 smithi175 ceph-mon[36128]: pgmap v998: 129 pgs: 129 active+clean; 231 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-09-13T15:58:36.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:36 smithi090 ceph-mon[27811]: pgmap v998: 129 pgs: 129 active+clean; 231 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-09-13T15:58:36.553 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:58:36.758 INFO:teuthology.orchestra.run.smithi090.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-13T15:58:36.773 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T15:58:36.773 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T15:58:36.843 INFO:teuthology.orchestra.run.smithi090.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T15:58:37.370 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T15:58:37.370 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:37.370 INFO:teuthology.orchestra.run.smithi175.stdout:Removed: 2024-09-13T15:58:37.370 INFO:teuthology.orchestra.run.smithi175.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T15:58:37.370 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:37.370 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:58:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-13T15:58:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-13T15:58:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-13T15:58:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:38.121 INFO:teuthology.orchestra.run.smithi090.stdout:Removed: 2024-09-13T15:58:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T15:58:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:38.122 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:58:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:38 smithi090 ceph-mon[27811]: pgmap v999: 129 pgs: 129 active+clean; 229 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-09-13T15:58:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:58:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:58:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:58:38.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:38 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:58:38.342 INFO:teuthology.orchestra.run.smithi175.stdout:Last metadata expiration check: 0:36:07 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:58:38.494 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-09-13T15:58:38.494 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: Package Architecture Version Repository Size 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-09-13T15:58:38.495 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-09-13T15:58:38.496 INFO:teuthology.orchestra.run.smithi175.stdout:Install 6 Packages 2024-09-13T15:58:38.496 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:38.511 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 2.0 M 2024-09-13T15:58:38.511 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 9.2 M 2024-09-13T15:58:38.511 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-09-13T15:58:38.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:38 smithi175 ceph-mon[36128]: pgmap v999: 129 pgs: 129 active+clean; 229 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.4 KiB/s rd, 1.3 MiB/s wr, 117 op/s 2024-09-13T15:58:38.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-13T15:58:38.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-13T15:58:38.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-13T15:58:38.517 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:38 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' 2024-09-13T15:58:38.736 INFO:teuthology.orchestra.run.smithi175.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 295 kB/s | 44 kB 00:00 2024-09-13T15:58:38.762 INFO:teuthology.orchestra.run.smithi175.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 603 kB/s | 106 kB 00:00 2024-09-13T15:58:38.820 INFO:teuthology.orchestra.run.smithi175.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 859 kB/s | 72 kB 00:00 2024-09-13T15:58:38.845 INFO:teuthology.orchestra.run.smithi175.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.0 MB/s | 84 kB 00:00 2024-09-13T15:58:38.920 INFO:teuthology.orchestra.run.smithi175.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.6 MB/s | 906 kB 00:00 2024-09-13T15:58:38.971 INFO:teuthology.orchestra.run.smithi175.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.4 MB/s | 837 kB 00:00 2024-09-13T15:58:38.971 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:58:38.971 INFO:teuthology.orchestra.run.smithi175.stdout:Total 4.4 MB/s | 2.0 MB 00:00 2024-09-13T15:58:39.060 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-09-13T15:58:39.101 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-09-13T15:58:39.101 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-09-13T15:58:39.169 INFO:teuthology.orchestra.run.smithi090.stdout:Last metadata expiration check: 0:36:08 ago on Fri 13 Sep 2024 03:22:31 PM UTC. 2024-09-13T15:58:39.328 INFO:teuthology.orchestra.run.smithi090.stdout:Dependencies resolved. 2024-09-13T15:58:39.328 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:58:39.328 INFO:teuthology.orchestra.run.smithi090.stdout: Package Architecture Version Repository Size 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout:Installing: 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout:Installing dependencies: 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction Summary 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout:================================================================================ 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout:Install 6 Packages 2024-09-13T15:58:39.329 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:39.375 INFO:teuthology.orchestra.run.smithi090.stdout:Total download size: 2.0 M 2024-09-13T15:58:39.376 INFO:teuthology.orchestra.run.smithi090.stdout:Installed size: 9.2 M 2024-09-13T15:58:39.376 INFO:teuthology.orchestra.run.smithi090.stdout:Downloading Packages: 2024-09-13T15:58:39.548 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-09-13T15:58:39.548 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-09-13T15:58:39.613 INFO:teuthology.orchestra.run.smithi090.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 311 kB/s | 44 kB 00:00 2024-09-13T15:58:39.655 INFO:teuthology.orchestra.run.smithi090.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 572 kB/s | 106 kB 00:00 2024-09-13T15:58:39.740 INFO:teuthology.orchestra.run.smithi090.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 564 kB/s | 72 kB 00:00 2024-09-13T15:58:39.773 INFO:teuthology.orchestra.run.smithi090.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 710 kB/s | 84 kB 00:00 2024-09-13T15:58:39.858 INFO:teuthology.orchestra.run.smithi090.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.3 MB/s | 906 kB 00:00 2024-09-13T15:58:39.933 INFO:teuthology.orchestra.run.smithi090.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.2 MB/s | 837 kB 00:00 2024-09-13T15:58:39.933 INFO:teuthology.orchestra.run.smithi090.stdout:-------------------------------------------------------------------------------- 2024-09-13T15:58:39.933 INFO:teuthology.orchestra.run.smithi090.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2024-09-13T15:58:39.974 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-09-13T15:58:40.031 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction check 2024-09-13T15:58:40.067 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction check succeeded. 2024-09-13T15:58:40.067 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction test 2024-09-13T15:58:40.107 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-13T15:58:40.158 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-13T15:58:40.203 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-13T15:58:40.417 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-13T15:58:40.477 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T15:58:40.491 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:40 smithi175 ceph-mon[36128]: pgmap v1000: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 797 KiB/s wr, 108 op/s 2024-09-13T15:58:40.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:40 smithi090 ceph-mon[27811]: pgmap v1000: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 9.6 KiB/s rd, 797 KiB/s wr, 108 op/s 2024-09-13T15:58:40.578 INFO:teuthology.orchestra.run.smithi090.stdout:Transaction test succeeded. 2024-09-13T15:58:40.579 INFO:teuthology.orchestra.run.smithi090.stdout:Running transaction 2024-09-13T15:58:41.030 INFO:teuthology.orchestra.run.smithi090.stdout: Preparing : 1/1 2024-09-13T15:58:41.094 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T15:58:41.129 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T15:58:41.171 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-13T15:58:41.239 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-13T15:58:41.287 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-13T15:58:41.507 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-13T15:58:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T15:58:42.175 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:42 smithi090 ceph-mon[27811]: pgmap v1001: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 81 KiB/s wr, 43 op/s 2024-09-13T15:58:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-13T15:58:42.228 INFO:teuthology.orchestra.run.smithi090.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T15:58:42.478 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T15:58:42.478 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-13T15:58:42.478 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-13T15:58:42.478 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-13T15:58:42.478 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-13T15:58:42.478 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-13T15:58:42.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:42 smithi175 ceph-mon[36128]: pgmap v1001: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 81 KiB/s wr, 43 op/s 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T15:58:42.925 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-09-13T15:58:42.926 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-09-13T15:58:43.298 DEBUG:teuthology.parallel:result is None 2024-09-13T15:58:43.798 INFO:teuthology.orchestra.run.smithi090.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-13T15:58:43.798 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-13T15:58:43.798 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-13T15:58:43.799 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-13T15:58:43.799 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-13T15:58:43.799 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout:Installed: 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:44.246 INFO:teuthology.orchestra.run.smithi090.stdout:Complete! 2024-09-13T15:58:44.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:44 smithi175 ceph-mon[36128]: pgmap v1002: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 17 KiB/s wr, 34 op/s 2024-09-13T15:58:44.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:44 smithi090 ceph-mon[27811]: pgmap v1002: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 938 B/s rd, 17 KiB/s wr, 34 op/s 2024-09-13T15:58:44.728 DEBUG:teuthology.parallel:result is None 2024-09-13T15:58:44.729 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-13T15:58:44.744 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-13T15:58:44.745 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:58:44.745 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:58:44.777 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:58:44.777 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:58:44.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:58:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 -- 172.21.15.90:0/3758538819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 msgr2=0x7fd3ac1030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/3758538819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac1030d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd39c009880 tx=0x7fd39c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.225 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 -- 172.21.15.90:0/3758538819 shutdown_connections 2024-09-13T15:58:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/3758538819 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac1030d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/3758538819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 0x7fd3ac100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 -- 172.21.15.90:0/3758538819 >> 172.21.15.90:0/3758538819 conn(0x7fd3ac0fa100 msgr2=0x7fd3ac0fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 -- 172.21.15.90:0/3758538819 shutdown_connections 2024-09-13T15:58:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.218+0000 7fd3b3c19640 1 -- 172.21.15.90:0/3758538819 wait complete. 2024-09-13T15:58:45.226 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b3c19640 1 Processor -- start 2024-09-13T15:58:45.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b3c19640 1 -- start start 2024-09-13T15:58:45.227 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b3c19640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 0x7fd3ac19b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b3c19640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac19bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.228 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b3c19640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac19c450 con 0x7fd3ac100c60 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b3c19640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac19c5c0 con 0x7fd3ac0fe310 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b11b4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac19bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b11b4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac19bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36540/0 (socket says 172.21.15.90:36540) 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b11b4640 1 -- 172.21.15.90:0/673275984 learned_addr learned my addr 172.21.15.90:0/673275984 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b19b5640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 0x7fd3ac19b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b11b4640 1 -- 172.21.15.90:0/673275984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 msgr2=0x7fd3ac19b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b11b4640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 0x7fd3ac19b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b11b4640 1 -- 172.21.15.90:0/673275984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c009560 con 0x7fd3ac100c60 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.219+0000 7fd3b19b5640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 0x7fd3ac19b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T15:58:45.229 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.220+0000 7fd3b11b4640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac19bf10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd39c004f60 tx=0x7fd39c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.220+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd39c005560 con 0x7fd3ac100c60 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.220+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac1a1030 con 0x7fd3ac100c60 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.220+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd39c003a80 con 0x7fd3ac100c60 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.220+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd39c03ca40 con 0x7fd3ac100c60 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.220+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac1a1500 con 0x7fd3ac100c60 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.221+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac10eaa0 con 0x7fd3ac100c60 2024-09-13T15:58:45.230 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.222+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fd39c003bf0 con 0x7fd3ac100c60 2024-09-13T15:58:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.222+0000 7fd3a2ffd640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd39806cc50 0x7fd39806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.222+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7fd39c0add20 con 0x7fd3ac100c60 2024-09-13T15:58:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.222+0000 7fd3b19b5640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd39806cc50 0x7fd39806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.223+0000 7fd3b19b5640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd39806cc50 0x7fd39806f110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005d00 tx=0x7fd3a8005c70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:45.231 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.227+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fd39c077910 con 0x7fd3ac100c60 2024-09-13T15:58:45.396 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.395+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3ac10ec80 con 0x7fd3ac100c60 2024-09-13T15:58:45.397 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.396+0000 7fd3a2ffd640 1 -- 172.21.15.90:0/673275984 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+21381 (secure 0 0 0) 0x7fd39c07af30 con 0x7fd3ac100c60 2024-09-13T15:58:45.398 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:45.398 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":72,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:19.566068+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.175:6832/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.175:6833/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:58:45.400 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd39806cc50 msgr2=0x7fd39806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd39806cc50 0x7fd39806f110 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8005d00 tx=0x7fd3a8005c70 comp rx=0 tx=0).stop 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 msgr2=0x7fd3ac19bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac19bf10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd39c004f60 tx=0x7fd39c004600 comp rx=0 tx=0).stop 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 shutdown_connections 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fd39806cc50 0x7fd39806f110 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fd3ac100c60 0x7fd3ac19bf10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 --2- 172.21.15.90:0/673275984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3ac0fe310 0x7fd3ac19b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 >> 172.21.15.90:0/673275984 conn(0x7fd3ac0fa100 msgr2=0x7fd3ac0fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 shutdown_connections 2024-09-13T15:58:45.401 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.399+0000 7fd3b3c19640 1 -- 172.21.15.90:0/673275984 wait complete. 2024-09-13T15:58:45.411 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:58:45.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.513+0000 7fb5c3533640 1 -- 172.21.15.90:0/1799074754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc1004f0 msgr2=0x7fb5bc1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.513+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1799074754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc1004f0 0x7fb5bc1028e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009880 tx=0x7fb5b802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- 172.21.15.90:0/1799074754 shutdown_connections 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1799074754 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5bc102e20 0x7fb5bc105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1799074754 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc1004f0 0x7fb5bc1028e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- 172.21.15.90:0/1799074754 >> 172.21.15.90:0/1799074754 conn(0x7fb5bc0fa0f0 msgr2=0x7fb5bc0fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:45.517 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- 172.21.15.90:0/1799074754 shutdown_connections 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- 172.21.15.90:0/1799074754 wait complete. 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 Processor -- start 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- start start 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5bc1004f0 0x7fb5bc1953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 0x7fb5bc195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc195e60 con 0x7fb5bc102e20 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c3533640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc195fd0 con 0x7fb5bc1004f0 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c0ace640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 0x7fb5bc195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c0ace640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 0x7fb5bc195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36554/0 (socket says 172.21.15.90:36554) 2024-09-13T15:58:45.518 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.514+0000 7fb5c0ace640 1 -- 172.21.15.90:0/1559327232 learned_addr learned my addr 172.21.15.90:0/1559327232 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5c0ace640 1 -- 172.21.15.90:0/1559327232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5bc1004f0 msgr2=0x7fb5bc1953e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5c0ace640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5bc1004f0 0x7fb5bc1953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5c0ace640 1 -- 172.21.15.90:0/1559327232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b8009560 con 0x7fb5bc102e20 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5c0ace640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 0x7fb5bc195920 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00c8f0 tx=0x7fb5ac00cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ac004410 con 0x7fb5bc102e20 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb5ac007c30 con 0x7fb5bc102e20 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc19aaa0 con 0x7fb5bc102e20 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5ac003d80 con 0x7fb5bc102e20 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.515+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc19b050 con 0x7fb5bc102e20 2024-09-13T15:58:45.519 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.516+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc1008d0 con 0x7fb5bc102e20 2024-09-13T15:58:45.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.516+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb5ac0077a0 con 0x7fb5bc102e20 2024-09-13T15:58:45.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.517+0000 7fb5b27fc640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a806cc50 0x7fb5a806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.517+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7fb5ac08c980 con 0x7fb5bc102e20 2024-09-13T15:58:45.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.517+0000 7fb5c12cf640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a806cc50 0x7fb5a806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.520 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.517+0000 7fb5c12cf640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a806cc50 0x7fb5a806f110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb5b802a9c0 tx=0x7fb5b8000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:45.522 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.521+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb5ac0565f0 con 0x7fb5bc102e20 2024-09-13T15:58:45.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.682+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5bc19b320 con 0x7fb5bc102e20 2024-09-13T15:58:45.683 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.682+0000 7fb5b27fc640 1 -- 172.21.15.90:0/1559327232 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5ac059c10 con 0x7fb5bc102e20 2024-09-13T15:58:45.684 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:58:45.685 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a806cc50 msgr2=0x7fb5a806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a806cc50 0x7fb5a806f110 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb5b802a9c0 tx=0x7fb5b8000bc0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 msgr2=0x7fb5bc195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 0x7fb5bc195920 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac00c8f0 tx=0x7fb5ac00cdb0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 shutdown_connections 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5a806cc50 0x7fb5a806f110 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5bc102e20 0x7fb5bc195920 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 --2- 172.21.15.90:0/1559327232 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5bc1004f0 0x7fb5bc1953e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 >> 172.21.15.90:0/1559327232 conn(0x7fb5bc0fa0f0 msgr2=0x7fb5bc0fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 shutdown_connections 2024-09-13T15:58:45.686 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.684+0000 7fb5c3533640 1 -- 172.21.15.90:0/1559327232 wait complete. 2024-09-13T15:58:45.698 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:58:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.801+0000 7f6193745640 1 -- 172.21.15.90:0/409641333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 msgr2=0x7f618c0fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.801+0000 7f6193745640 1 --2- 172.21.15.90:0/409641333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c0fffe0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f617c009880 tx=0x7f617c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.805 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.801+0000 7f6193745640 1 -- 172.21.15.90:0/409641333 shutdown_connections 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.801+0000 7f6193745640 1 --2- 172.21.15.90:0/409641333 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c0fffe0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 --2- 172.21.15.90:0/409641333 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f618c0fe970 0x7f618c0fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 -- 172.21.15.90:0/409641333 >> 172.21.15.90:0/409641333 conn(0x7f618c0fa0f0 msgr2=0x7f618c0fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 -- 172.21.15.90:0/409641333 shutdown_connections 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 -- 172.21.15.90:0/409641333 wait complete. 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 Processor -- start 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 -- start start 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f618c0fe970 0x7f618c067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.806 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c19a670 con 0x7f618c0ffb60 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6193745640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618c19a7b0 con 0x7f618c0fe970 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6190ce0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6190ce0640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36566/0 (socket says 172.21.15.90:36566) 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.802+0000 7f6190ce0640 1 -- 172.21.15.90:0/1300129547 learned_addr learned my addr 172.21.15.90:0/1300129547 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f61914e1640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f618c0fe970 0x7f618c067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f6190ce0640 1 -- 172.21.15.90:0/1300129547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f618c0fe970 msgr2=0x7f618c067fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f6190ce0640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f618c0fe970 0x7f618c067fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f6190ce0640 1 -- 172.21.15.90:0/1300129547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c009560 con 0x7f618c0ffb60 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f6190ce0640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c19a130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f617c004f60 tx=0x7f617c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617c005560 con 0x7f618c0ffb60 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f617c003a80 con 0x7f618c0ffb60 2024-09-13T15:58:45.807 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618c19aa00 con 0x7f618c0ffb60 2024-09-13T15:58:45.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f617c03ca40 con 0x7f618c0ffb60 2024-09-13T15:58:45.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.803+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f618c19af80 con 0x7f618c0ffb60 2024-09-13T15:58:45.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.804+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618c1073a0 con 0x7f618c0ffb60 2024-09-13T15:58:45.808 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.805+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f617c003bf0 con 0x7f618c0ffb60 2024-09-13T15:58:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.805+0000 7f61827fc640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f617806cc50 0x7f617806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.805+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f617c0adca0 con 0x7f618c0ffb60 2024-09-13T15:58:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.805+0000 7f61914e1640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f617806cc50 0x7f617806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:45.809 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.805+0000 7f61914e1640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f617806cc50 0x7f617806f110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f618c0ff9c0 tx=0x7f6188009680 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:45.810 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.809+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f617c0b2070 con 0x7f618c0ffb60 2024-09-13T15:58:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.977+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7f618c19b400 con 0x7f618c0ffb60 2024-09-13T15:58:45.979 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.978+0000 7f61827fc640 1 -- 172.21.15.90:0/1300129547 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f617c077940 con 0x7f618c0ffb60 2024-09-13T15:58:45.979 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f617806cc50 msgr2=0x7f617806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f617806cc50 0x7f617806f110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f618c0ff9c0 tx=0x7f6188009680 comp rx=0 tx=0).stop 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 msgr2=0x7f618c19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c19a130 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f617c004f60 tx=0x7f617c004600 comp rx=0 tx=0).stop 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 shutdown_connections 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f617806cc50 0x7f617806f110 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f618c0ffb60 0x7f618c19a130 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 --2- 172.21.15.90:0/1300129547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f618c0fe970 0x7f618c067fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 >> 172.21.15.90:0/1300129547 conn(0x7f618c0fa0f0 msgr2=0x7f618c102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 shutdown_connections 2024-09-13T15:58:45.982 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:45.980+0000 7f6193745640 1 -- 172.21.15.90:0/1300129547 wait complete. 2024-09-13T15:58:45.992 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 -- 172.21.15.90:0/1339945902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 msgr2=0x7fae2c0ffe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 --2- 172.21.15.90:0/1339945902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c0ffe90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fae20009880 tx=0x7fae2002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 -- 172.21.15.90:0/1339945902 shutdown_connections 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 --2- 172.21.15.90:0/1339945902 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae2c100c80 0x7fae2c101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 --2- 172.21.15.90:0/1339945902 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c0ffe90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 -- 172.21.15.90:0/1339945902 >> 172.21.15.90:0/1339945902 conn(0x7fae2c0fb1f0 msgr2=0x7fae2c0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.095 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 -- 172.21.15.90:0/1339945902 shutdown_connections 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.092+0000 7fae3325a640 1 -- 172.21.15.90:0/1339945902 wait complete. 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae3325a640 1 Processor -- start 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae3325a640 1 -- start start 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae3325a640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae3325a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae2c100c80 0x7fae2c195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae3325a640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c195ff0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae3325a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae2c196160 con 0x7fae2c100c80 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36578/0 (socket says 172.21.15.90:36578) 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 -- 172.21.15.90:0/4063029442 learned_addr learned my addr 172.21.15.90:0/4063029442 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae2bfff640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae2c100c80 0x7fae2c195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.096 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 -- 172.21.15.90:0/4063029442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae2c100c80 msgr2=0x7fae2c195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae2c100c80 0x7fae2c195ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 -- 172.21.15.90:0/4063029442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20009560 con 0x7fae2c0ffa90 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.093+0000 7fae30ff6640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c195570 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fae20006010 tx=0x7fae20004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.094+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae20005560 con 0x7fae2c0ffa90 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.094+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c19abd0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.094+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fae20003a80 con 0x7fae2c0ffa90 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.094+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae2003c9e0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.097 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.094+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c19b0a0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.095+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c0fff10 con 0x7fae2c0ffa90 2024-09-13T15:58:46.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.095+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fae20003bf0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.096+0000 7fae29ffb640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fae1c06cc50 0x7fae1c06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.096+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7fae200adcf0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.096+0000 7fae2bfff640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fae1c06cc50 0x7fae1c06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.098 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.096+0000 7fae2bfff640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fae1c06cc50 0x7fae1c06f110 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fae2c196a80 tx=0x7fae18008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.101 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.100+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fae200778e0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:46 smithi090 ceph-mon[27811]: pgmap v1003: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 2 op/s 2024-09-13T15:58:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/673275984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:58:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1559327232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:58:46.251 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:46 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1300129547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.265+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7fae2c19b2c0 con 0x7fae2c0ffa90 2024-09-13T15:58:46.267 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.266+0000 7fae29ffb640 1 -- 172.21.15.90:0/4063029442 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae2007af00 con 0x7fae2c0ffa90 2024-09-13T15:58:46.267 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.267+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fae1c06cc50 msgr2=0x7fae1c06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fae1c06cc50 0x7fae1c06f110 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fae2c196a80 tx=0x7fae18008040 comp rx=0 tx=0).stop 2024-09-13T15:58:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 msgr2=0x7fae2c195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c195570 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fae20006010 tx=0x7fae20004da0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.269 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 shutdown_connections 2024-09-13T15:58:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fae1c06cc50 0x7fae1c06f110 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae2c100c80 0x7fae2c195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 --2- 172.21.15.90:0/4063029442 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fae2c0ffa90 0x7fae2c195570 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 >> 172.21.15.90:0/4063029442 conn(0x7fae2c0fb1f0 msgr2=0x7fae2c103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 shutdown_connections 2024-09-13T15:58:46.270 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.268+0000 7fae3325a640 1 -- 172.21.15.90:0/4063029442 wait complete. 2024-09-13T15:58:46.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:58:46.338 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:46 smithi175 ceph-mon[36128]: pgmap v1003: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 2 op/s 2024-09-13T15:58:46.338 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/673275984' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:58:46.338 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1559327232' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:58:46.339 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:46 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1300129547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 -- 172.21.15.90:0/3914793591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 msgr2=0x7f4858100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 --2- 172.21.15.90:0/3914793591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f4858100010 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4840009880 tx=0x7f484002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 -- 172.21.15.90:0/3914793591 shutdown_connections 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 --2- 172.21.15.90:0/3914793591 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f4858100010 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 --2- 172.21.15.90:0/3914793591 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48580fe9a0 0x7f48580feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 -- 172.21.15.90:0/3914793591 >> 172.21.15.90:0/3914793591 conn(0x7f48580fa100 msgr2=0x7f48580fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 -- 172.21.15.90:0/3914793591 shutdown_connections 2024-09-13T15:58:46.385 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 -- 172.21.15.90:0/3914793591 wait complete. 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 Processor -- start 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.379+0000 7f485d31f640 1 -- start start 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f485d31f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48580fe9a0 0x7f4858068ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f485d31f640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f485819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f485d31f640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485819a710 con 0x7f48580ffb90 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f485d31f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485819a850 con 0x7f48580fe9a0 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f485819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f485819a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36608/0 (socket says 172.21.15.90:36608) 2024-09-13T15:58:46.386 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 -- 172.21.15.90:0/1309457270 learned_addr learned my addr 172.21.15.90:0/1309457270 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f4856ffd640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48580fe9a0 0x7f4858068ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 -- 172.21.15.90:0/1309457270 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48580fe9a0 msgr2=0x7f4858068ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48580fe9a0 0x7f4858068ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 -- 172.21.15.90:0/1309457270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4840009560 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f48567fc640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f485819a1d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4840004f60 tx=0x7f4840004da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48400054e0 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f485819aaa0 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4840005640 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f484003c960 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.380+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485819b020 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.381+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48581073d0 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.382+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f4840003cf0 con 0x7f48580ffb90 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.382+0000 7f484ffff640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f484406cc50 0x7f484406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.387 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.383+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f48400adc60 con 0x7f48580ffb90 2024-09-13T15:58:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.383+0000 7f4856ffd640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f484406cc50 0x7f484406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.383+0000 7f4856ffd640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f484406cc50 0x7f484406f110 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4848005fd0 tx=0x7f4848005850 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.388 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.387+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f4840077850 con 0x7f48580ffb90 2024-09-13T15:58:46.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.548+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7f485819b480 con 0x7f48580ffb90 2024-09-13T15:58:46.550 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.549+0000 7f484ffff640 1 -- 172.21.15.90:0/1309457270 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f484007ae70 con 0x7f48580ffb90 2024-09-13T15:58:46.550 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:46.552 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f484406cc50 msgr2=0x7f484406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f484406cc50 0x7f484406f110 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4848005fd0 tx=0x7f4848005850 comp rx=0 tx=0).stop 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 msgr2=0x7f485819a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f485819a1d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4840004f60 tx=0x7f4840004da0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 shutdown_connections 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f484406cc50 0x7f484406f110 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f48580ffb90 0x7f485819a1d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 --2- 172.21.15.90:0/1309457270 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f48580fe9a0 0x7f4858068ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 >> 172.21.15.90:0/1309457270 conn(0x7f48580fa100 msgr2=0x7f4858102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 shutdown_connections 2024-09-13T15:58:46.553 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.551+0000 7f485d31f640 1 -- 172.21.15.90:0/1309457270 wait complete. 2024-09-13T15:58:46.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:58:46.666 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- 172.21.15.90:0/3426606067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 msgr2=0x7f03541003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 --2- 172.21.15.90:0/3426606067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f03541003c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0348009880 tx=0x7f034802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- 172.21.15.90:0/3426606067 shutdown_connections 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 --2- 172.21.15.90:0/3426606067 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03540fec10 0x7f03540ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 --2- 172.21.15.90:0/3426606067 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f03541003c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- 172.21.15.90:0/3426606067 >> 172.21.15.90:0/3426606067 conn(0x7f03540f9890 msgr2=0x7f03540fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- 172.21.15.90:0/3426606067 shutdown_connections 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- 172.21.15.90:0/3426606067 wait complete. 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 Processor -- start 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- start start 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03540fec10 0x7f0354195430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.667 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f0354195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0354195eb0 con 0x7f03540fffc0 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.664+0000 7f035ac79640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0354196020 con 0x7f03540fec10 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f0354195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f0354195970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36622/0 (socket says 172.21.15.90:36622) 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 -- 172.21.15.90:0/4202433903 learned_addr learned my addr 172.21.15.90:0/4202433903 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0358a15640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03540fec10 0x7f0354195430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 -- 172.21.15.90:0/4202433903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03540fec10 msgr2=0x7f0354195430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03540fec10 0x7f0354195430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 -- 172.21.15.90:0/4202433903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0348009560 con 0x7f03540fffc0 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0353fff640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f0354195970 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f034000c8f0 tx=0x7f034000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0340007b80 con 0x7f03540fffc0 2024-09-13T15:58:46.668 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0340007ce0 con 0x7f03540fffc0 2024-09-13T15:58:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035419aaf0 con 0x7f03540fffc0 2024-09-13T15:58:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0340018630 con 0x7f03540fffc0 2024-09-13T15:58:46.669 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.665+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035419b0a0 con 0x7f03540fffc0 2024-09-13T15:58:46.670 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.666+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03540ff090 con 0x7f03540fffc0 2024-09-13T15:58:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.667+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f034000f450 con 0x7f03540fffc0 2024-09-13T15:58:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.667+0000 7f0351ffb640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f034406cc50 0x7f034406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.667+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f034008cc50 con 0x7f03540fffc0 2024-09-13T15:58:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.667+0000 7f0358a15640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f034406cc50 0x7f034406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.671 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.668+0000 7f0358a15640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f034406cc50 0x7f034406f110 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0348000c00 tx=0x7f0348004f00 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.672 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.671+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f0340056970 con 0x7f03540fffc0 2024-09-13T15:58:46.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.833+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7f035419b2f0 con 0x7f03540fffc0 2024-09-13T15:58:46.834 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.833+0000 7f0351ffb640 1 -- 172.21.15.90:0/4202433903 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f034000f700 con 0x7f03540fffc0 2024-09-13T15:58:46.835 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:46.836 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f034406cc50 msgr2=0x7f034406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f034406cc50 0x7f034406f110 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0348000c00 tx=0x7f0348004f00 comp rx=0 tx=0).stop 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 msgr2=0x7f0354195970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f0354195970 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f034000c8f0 tx=0x7f034000cdb0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 shutdown_connections 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f034406cc50 0x7f034406f110 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f03540fffc0 0x7f0354195970 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 --2- 172.21.15.90:0/4202433903 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f03540fec10 0x7f0354195430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 >> 172.21.15.90:0/4202433903 conn(0x7f03540f9890 msgr2=0x7f03540fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 shutdown_connections 2024-09-13T15:58:46.837 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.835+0000 7f035ac79640 1 -- 172.21.15.90:0/4202433903 wait complete. 2024-09-13T15:58:46.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.947+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/2362280558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 msgr2=0x7fb5f00fed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.947+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/2362280558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f00fed90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4009880 tx=0x7fb5e402a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/2362280558 shutdown_connections 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/2362280558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5f00ffb80 0x7fb5f0100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/2362280558 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f00fed90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/2362280558 >> 172.21.15.90:0/2362280558 conn(0x7fb5f00fa0f0 msgr2=0x7fb5f00fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/2362280558 shutdown_connections 2024-09-13T15:58:46.952 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/2362280558 wait complete. 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 Processor -- start 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.948+0000 7fb5f6bf4640 1 -- start start 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f6bf4640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f0067fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f6bf4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5f00ffb80 0x7fb5f019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f6bf4640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f019a630 con 0x7fb5f00fe990 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f6bf4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f019a770 con 0x7fb5f00ffb80 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f0067fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f0067fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36642/0 (socket says 172.21.15.90:36642) 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 -- 172.21.15.90:0/1211563821 learned_addr learned my addr 172.21.15.90:0/1211563821 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5effff640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5f00ffb80 0x7fb5f019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 -- 172.21.15.90:0/1211563821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5f00ffb80 msgr2=0x7fb5f019a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:46.953 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5f00ffb80 0x7fb5f019a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 -- 172.21.15.90:0/1211563821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4009560 con 0x7fb5f00fe990 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f4990640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f0067fc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4006010 tx=0x7fb5e4005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5e4005600 con 0x7fb5f00fe990 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f019a9c0 con 0x7fb5f00fe990 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb5e4003a80 con 0x7fb5f00fe990 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.949+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5e403ca20 con 0x7fb5f00fe990 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.950+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f019af40 con 0x7fb5f00fe990 2024-09-13T15:58:46.954 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.951+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f01073a0 con 0x7fb5f00fe990 2024-09-13T15:58:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.951+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fb5e403cb80 con 0x7fb5f00fe990 2024-09-13T15:58:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.951+0000 7fb5edffb640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5e006cc50 0x7fb5e006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.952+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7fb5e40adcd0 con 0x7fb5f00fe990 2024-09-13T15:58:46.955 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.952+0000 7fb5effff640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5e006cc50 0x7fb5e006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:46.956 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.952+0000 7fb5effff640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5e006cc50 0x7fb5e006f110 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0067d30 tx=0x7fb5dc00a2d0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:46.957 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:46.956+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fb5e4077930 con 0x7fb5f00fe990 2024-09-13T15:58:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.137+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb5f0107530 con 0x7fb5f00fe990 2024-09-13T15:58:47.138 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.137+0000 7fb5edffb640 1 -- 172.21.15.90:0/1211563821 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v72) v1 ==== 81+0+1010 (secure 0 0 0) 0x7fb5e407af50 con 0x7fb5f00fe990 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6832/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6833/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3168219877 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3860788333 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/575512074 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:47.139 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2223577157 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/782157526 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1586604027 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3782750394 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2606530289 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/4162862940 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:47.140 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:listed 17 entries 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5e006cc50 msgr2=0x7fb5e006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5e006cc50 0x7fb5e006f110 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb5f0067d30 tx=0x7fb5dc00a2d0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 msgr2=0x7fb5f0067fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f0067fc0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4006010 tx=0x7fb5e4005000 comp rx=0 tx=0).stop 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 shutdown_connections 2024-09-13T15:58:47.141 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fb5e006cc50 0x7fb5e006f110 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5f00ffb80 0x7fb5f019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 --2- 172.21.15.90:0/1211563821 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fb5f00fe990 0x7fb5f0067fc0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 >> 172.21.15.90:0/1211563821 conn(0x7fb5f00fa0f0 msgr2=0x7fb5f0102db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 shutdown_connections 2024-09-13T15:58:47.142 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.139+0000 7fb5f6bf4640 1 -- 172.21.15.90:0/1211563821 wait complete. 2024-09-13T15:58:47.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:47 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/4063029442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:47.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:47 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1309457270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:47.150 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:47 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/4202433903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:47.151 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-09-13T15:58:47.151 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:58:47.172 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T15:58:47.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 -- 172.21.15.90:0/1347339557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0ff2d0 msgr2=0x7f74fc0ff750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 --2- 172.21.15.90:0/1347339557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0ff2d0 0x7f74fc0ff750 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00a990 tx=0x7f74ec02b7d0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 -- 172.21.15.90:0/1347339557 shutdown_connections 2024-09-13T15:58:47.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 --2- 172.21.15.90:0/1347339557 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0ff2d0 0x7f74fc0ff750 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.274 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 --2- 172.21.15.90:0/1347339557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc0fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 -- 172.21.15.90:0/1347339557 >> 172.21.15.90:0/1347339557 conn(0x7f74fc0f9430 msgr2=0x7f74fc0fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 -- 172.21.15.90:0/1347339557 shutdown_connections 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.271+0000 7f75009bc640 1 -- 172.21.15.90:0/1347339557 wait complete. 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f75009bc640 1 Processor -- start 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f75009bc640 1 -- start start 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f75009bc640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f75009bc640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74fc0ff2d0 0x7f74fc1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f75009bc640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc195ee0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f75009bc640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc196050 con 0x7f74fc0ff2d0 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f74fa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f74fa59c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc195460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36668/0 (socket says 172.21.15.90:36668) 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f74fa59c640 1 -- 172.21.15.90:0/681811062 learned_addr learned my addr 172.21.15.90:0/681811062 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:47.275 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f74fa59c640 1 -- 172.21.15.90:0/681811062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74fc0ff2d0 msgr2=0x7f74fc1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f74fa59c640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74fc0ff2d0 0x7f74fc1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.272+0000 7f74fa59c640 1 -- 172.21.15.90:0/681811062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec00a670 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.273+0000 7f74fa59c640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc195460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f74f000d8d0 tx=0x7f74f000dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.273+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f0009a30 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.273+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f74f0009b90 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.273+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74fc19aad0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.273+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f74f000f5f0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.276 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.273+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc19b080 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.274+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc0fe6e0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.274+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f74f0009d00 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.275+0000 7f74eb7fe640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f74e406cc50 0x7f74e406f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.275+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f74f008c8b0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.277 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.275+0000 7f74f9d9b640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f74e406cc50 0x7f74e406f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.278 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.275+0000 7f74f9d9b640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f74e406cc50 0x7f74e406f110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74ec036040 tx=0x7f74ec000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:47.280 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.279+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f74f00564a0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.444 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.443+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74fc18f170 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.446 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.445+0000 7f74eb7fe640 1 -- 172.21.15.90:0/681811062 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+21381 (secure 0 0 0) 0x7f74f0059ac0 con 0x7f74fc0fe2e0 2024-09-13T15:58:47.446 INFO:teuthology.orchestra.run.smithi090.stdout: 2024-09-13T15:58:47.446 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":72,"fsid":"41124e1c-71e4-11ef-bceb-c7b262605968","created":"2024-09-13T15:25:05.516587+0000","modified":"2024-09-13T15:33:19.566068+0000","last_up_change":"2024-09-13T15:31:31.343806+0000","last_in_change":"2024-09-13T15:31:14.579571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T15:29:32.801686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs.foo.meta","create_time":"2024-09-13T15:32:51.020685+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.data","create_time":"2024-09-13T15:32:51.863878+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.bar.meta","create_time":"2024-09-13T15:32:57.007077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"bar"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs.bar.data","create_time":"2024-09-13T15:32:57.978868+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"bar"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"358fe0ea-a41d-4789-903d-7e4613b4a0da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6803","nonce":679705213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6805","nonce":679705213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6809","nonce":679705213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":679705213},{"type":"v1","addr":"172.21.15.90:6807","nonce":679705213}]},"public_addr":"172.21.15.90:6803/679705213","cluster_addr":"172.21.15.90:6805/679705213","heartbeat_back_addr":"172.21.15.90:6809/679705213","heartbeat_front_addr":"172.21.15.90:6807/679705213","state":["exists","up"]},{"osd":1,"uuid":"9260e3c1-8e73-4a12-8caf-f5f664fd7c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6810","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6811","nonce":3388650116}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6813","nonce":3388650116}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6817","nonce":3388650116}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6814","nonce":3388650116},{"type":"v1","addr":"172.21.15.90:6815","nonce":3388650116}]},"public_addr":"172.21.15.90:6811/3388650116","cluster_addr":"172.21.15.90:6813/3388650116","heartbeat_back_addr":"172.21.15.90:6817/3388650116","heartbeat_front_addr":"172.21.15.90:6815/3388650116","state":["exists","up"]},{"osd":2,"uuid":"b19df8cf-75da-4134-badb-cdd4eb29c5db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6819","nonce":3155836791}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6820","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6821","nonce":3155836791}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6825","nonce":3155836791}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":3155836791},{"type":"v1","addr":"172.21.15.90:6823","nonce":3155836791}]},"public_addr":"172.21.15.90:6819/3155836791","cluster_addr":"172.21.15.90:6821/3155836791","heartbeat_back_addr":"172.21.15.90:6825/3155836791","heartbeat_front_addr":"172.21.15.90:6823/3155836791","state":["exists","up"]},{"osd":3,"uuid":"930bd74f-868e-4916-b6aa-d1729d711c5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6827","nonce":1255290974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6829","nonce":1255290974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6833","nonce":1255290974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":1255290974},{"type":"v1","addr":"172.21.15.90:6831","nonce":1255290974}]},"public_addr":"172.21.15.90:6827/1255290974","cluster_addr":"172.21.15.90:6829/1255290974","heartbeat_back_addr":"172.21.15.90:6833/1255290974","heartbeat_front_addr":"172.21.15.90:6831/1255290974","state":["exists","up"]},{"osd":4,"uuid":"488b1c67-90e5-4c01-b5f8-775fea197fb4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6801","nonce":2031407459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6803","nonce":2031407459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6807","nonce":2031407459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":2031407459},{"type":"v1","addr":"172.21.15.175:6805","nonce":2031407459}]},"public_addr":"172.21.15.175:6801/2031407459","cluster_addr":"172.21.15.175:6803/2031407459","heartbeat_back_addr":"172.21.15.175:6807/2031407459","heartbeat_front_addr":"172.21.15.175:6805/2031407459","state":["exists","up"]},{"osd":5,"uuid":"1a6e49d9-f113-4f29-9905-ebc5884e3471","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6809","nonce":1093936087}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6811","nonce":1093936087}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6815","nonce":1093936087}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1093936087},{"type":"v1","addr":"172.21.15.175:6813","nonce":1093936087}]},"public_addr":"172.21.15.175:6809/1093936087","cluster_addr":"172.21.15.175:6811/1093936087","heartbeat_back_addr":"172.21.15.175:6815/1093936087","heartbeat_front_addr":"172.21.15.175:6813/1093936087","state":["exists","up"]},{"osd":6,"uuid":"c7cfc7b4-d18f-4ce1-aff0-df8276d91ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6817","nonce":1206614294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6819","nonce":1206614294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6823","nonce":1206614294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":1206614294},{"type":"v1","addr":"172.21.15.175:6821","nonce":1206614294}]},"public_addr":"172.21.15.175:6817/1206614294","cluster_addr":"172.21.15.175:6819/1206614294","heartbeat_back_addr":"172.21.15.175:6823/1206614294","heartbeat_front_addr":"172.21.15.175:6821/1206614294","state":["exists","up"]},{"osd":7,"uuid":"325ddbd7-1fe4-494f-a513-865f12961c78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6824","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6825","nonce":1677596558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6827","nonce":1677596558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6831","nonce":1677596558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":1677596558},{"type":"v1","addr":"172.21.15.175:6829","nonce":1677596558}]},"public_addr":"172.21.15.175:6825/1677596558","cluster_addr":"172.21.15.175:6827/1677596558","heartbeat_back_addr":"172.21.15.175:6831/1677596558","heartbeat_front_addr":"172.21.15.175:6829/1677596558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:28:35.271783+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:01.459956+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:28.733105+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:29:55.068914+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:16.335981+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:30:40.985957+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:05.202309+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T15:31:29.644154+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":7,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.175:6832/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.175:6833/2042693132":"2024-09-14T15:33:06.895655+0000","172.21.15.90:0/3168219877":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/3860788333":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/575512074":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6800/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6801/1322912193":"2024-09-14T15:25:26.727266+0000","172.21.15.90:6800/3319538530":"2024-09-14T15:27:18.494403+0000","172.21.15.90:0/2223577157":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/782157526":"2024-09-14T15:25:26.727266+0000","172.21.15.90:0/1586604027":"2024-09-14T15:26:07.051031+0000","172.21.15.90:6801/3565908774":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/3782750394":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/2606530289":"2024-09-14T15:26:07.051031+0000","172.21.15.90:0/4162862940":"2024-09-14T15:27:18.494403+0000","172.21.15.90:6801/3319538530":"2024-09-14T15:27:18.494403+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T15:58:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f74e406cc50 msgr2=0x7f74e406f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f74e406cc50 0x7f74e406f110 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f74ec036040 tx=0x7f74ec000bc0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 msgr2=0x7f74fc195460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc195460 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f74f000d8d0 tx=0x7f74f000dd90 comp rx=0 tx=0).stop 2024-09-13T15:58:47.448 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 shutdown_connections 2024-09-13T15:58:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f74e406cc50 0x7f74e406f110 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74fc0ff2d0 0x7f74fc1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 --2- 172.21.15.90:0/681811062 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f74fc0fe2e0 0x7f74fc195460 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 >> 172.21.15.90:0/681811062 conn(0x7f74fc0f9430 msgr2=0x7f74fc0fb180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 shutdown_connections 2024-09-13T15:58:47.449 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.447+0000 7f75009bc640 1 -- 172.21.15.90:0/681811062 wait complete. 2024-09-13T15:58:47.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T15:29:32.801686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs.foo.meta', 'create_time': '2024-09-13T15:32:51.020685+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs.foo.data', 'create_time': '2024-09-13T15:32:51.863878+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'foo'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs.bar.meta', 'create_time': '2024-09-13T15:32:57.007077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'bar'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs.bar.data', 'create_time': '2024-09-13T15:32:57.978868+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'bar'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T15:58:47.459 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T15:58:47.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:47 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/4063029442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:47.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:47 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1309457270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:47.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:47 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/4202433903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.559+0000 7fe80c3ce640 1 -- 172.21.15.90:0/2017836007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 msgr2=0x7fe8040fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.559+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/2017836007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe8040fffe0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009880 tx=0x7fe7f802a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- 172.21.15.90:0/2017836007 shutdown_connections 2024-09-13T15:58:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/2017836007 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe8040fffe0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/2017836007 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 0x7fe8040fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.563 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- 172.21.15.90:0/2017836007 >> 172.21.15.90:0/2017836007 conn(0x7fe8040fa0f0 msgr2=0x7fe8040fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- 172.21.15.90:0/2017836007 shutdown_connections 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- 172.21.15.90:0/2017836007 wait complete. 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 Processor -- start 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- start start 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 0x7fe804102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe804100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe804102e40 con 0x7fe8040ffb60 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe80c3ce640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8041014b0 con 0x7fe8040fe970 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe809969640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe804100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe809969640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe804100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36688/0 (socket says 172.21.15.90:36688) 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.560+0000 7fe809969640 1 -- 172.21.15.90:0/3639477296 learned_addr learned my addr 172.21.15.90:0/3639477296 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe80a16a640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 0x7fe804102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.564 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe809969640 1 -- 172.21.15.90:0/3639477296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 msgr2=0x7fe804102900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe809969640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 0x7fe804102900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe809969640 1 -- 172.21.15.90:0/3639477296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f8009560 con 0x7fe8040ffb60 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe80a16a640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 0x7fe804102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe809969640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe804100f40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8005940 tx=0x7fe7f8005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7f8005630 con 0x7fe8040ffb60 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe7f8003a80 con 0x7fe8040ffb60 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe7f803cb20 con 0x7fe8040ffb60 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe804101730 con 0x7fe8040ffb60 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.561+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe804101c30 con 0x7fe8040ffb60 2024-09-13T15:58:47.565 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.562+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe804107420 con 0x7fe8040ffb60 2024-09-13T15:58:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.563+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7fe7f8003bf0 con 0x7fe8040ffb60 2024-09-13T15:58:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.563+0000 7fe8037fe640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe7ec06cc50 0x7fe7ec06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.563+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7fe7f80adec0 con 0x7fe8040ffb60 2024-09-13T15:58:47.566 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.563+0000 7fe80a16a640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe7ec06cc50 0x7fe7ec06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.567 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.563+0000 7fe80a16a640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe7ec06cc50 0x7fe7ec06f110 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe8040ff9c0 tx=0x7fe7f40073c0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:47.568 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.567+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7fe7f8077ab0 con 0x7fe8040ffb60 2024-09-13T15:58:47.729 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.728+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe80419d7e0 con 0x7fe8040ffb60 2024-09-13T15:58:47.730 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.728+0000 7fe8037fe640 1 -- 172.21.15.90:0/3639477296 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7f807b0d0 con 0x7fe8040ffb60 2024-09-13T15:58:47.730 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 1 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe7ec06cc50 msgr2=0x7fe7ec06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe7ec06cc50 0x7fe7ec06f110 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe8040ff9c0 tx=0x7fe7f40073c0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 msgr2=0x7fe804100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe804100f40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8005940 tx=0x7fe7f8005230 comp rx=0 tx=0).stop 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 shutdown_connections 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7fe7ec06cc50 0x7fe7ec06f110 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7fe8040ffb60 0x7fe804100f40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.732 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 --2- 172.21.15.90:0/3639477296 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe8040fe970 0x7fe804102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.730+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 >> 172.21.15.90:0/3639477296 conn(0x7fe8040fa0f0 msgr2=0x7fe804067450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:47.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.731+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 shutdown_connections 2024-09-13T15:58:47.733 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.731+0000 7fe80c3ce640 1 -- 172.21.15.90:0/3639477296 wait complete. 2024-09-13T15:58:47.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.meta pg_num 2024-09-13T15:58:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 -- 172.21.15.90:0/2649272193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00ffb90 msgr2=0x7f3de0100010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 --2- 172.21.15.90:0/2649272193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00ffb90 0x7f3de0100010 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009c20 tx=0x7f3dc802a970 comp rx=0 tx=0).stop 2024-09-13T15:58:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 -- 172.21.15.90:0/2649272193 shutdown_connections 2024-09-13T15:58:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 --2- 172.21.15.90:0/2649272193 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00ffb90 0x7f3de0100010 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 --2- 172.21.15.90:0/2649272193 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3de00fe9a0 0x7f3de00feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.846 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 -- 172.21.15.90:0/2649272193 >> 172.21.15.90:0/2649272193 conn(0x7f3de00fa100 msgr2=0x7f3de00fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 -- 172.21.15.90:0/2649272193 shutdown_connections 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 -- 172.21.15.90:0/2649272193 wait complete. 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.843+0000 7f3de7133640 1 Processor -- start 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de7133640 1 -- start start 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de7133640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 0x7f3de0195590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de7133640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3de00ffb90 0x7f3de0195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de7133640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de0196010 con 0x7f3de00fe9a0 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de7133640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de0196180 con 0x7f3de00ffb90 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 0x7f3de0195590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 0x7f3de0195590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36708/0 (socket says 172.21.15.90:36708) 2024-09-13T15:58:47.847 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 -- 172.21.15.90:0/1470747124 learned_addr learned my addr 172.21.15.90:0/1470747124 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3ddffff640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3de00ffb90 0x7f3de0195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 -- 172.21.15.90:0/1470747124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3de00ffb90 msgr2=0x7f3de0195ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3de00ffb90 0x7f3de0195ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 -- 172.21.15.90:0/1470747124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc8009900 con 0x7f3de00fe9a0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de4ecf640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 0x7f3de0195590 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800b3d0 tx=0x7f3dd800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd8011800 con 0x7f3de00fe9a0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3dd8011df0 con 0x7f3de00fe9a0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de019ac50 con 0x7f3de00fe9a0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.844+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dd800f2e0 con 0x7f3de00fe9a0 2024-09-13T15:58:47.848 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.845+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de019b180 con 0x7f3de00fe9a0 2024-09-13T15:58:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.845+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de00fee20 con 0x7f3de00fe9a0 2024-09-13T15:58:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.846+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f3dd8011960 con 0x7f3de00fe9a0 2024-09-13T15:58:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.846+0000 7f3dddffb640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3dcc06cc50 0x7f3dcc06f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.846+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f3dd808c720 con 0x7f3de00fe9a0 2024-09-13T15:58:47.849 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.846+0000 7f3ddffff640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3dcc06cc50 0x7f3dcc06f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:47.850 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.847+0000 7f3ddffff640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3dcc06cc50 0x7f3dcc06f110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3dc802fbc0 tx=0x7f3dc8009800 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:47.851 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:47.851+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f3dd8056310 con 0x7f3de00fe9a0 2024-09-13T15:58:48.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.013+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"} v 0) v1 -- 0x7f3de019b3d0 con 0x7f3de00fe9a0 2024-09-13T15:58:48.014 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.013+0000 7f3dddffb640 1 -- 172.21.15.90:0/1470747124 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3dd8059930 con 0x7f3de00fe9a0 2024-09-13T15:58:48.015 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3dcc06cc50 msgr2=0x7f3dcc06f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3dcc06cc50 0x7f3dcc06f110 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3dc802fbc0 tx=0x7f3dc8009800 comp rx=0 tx=0).stop 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 msgr2=0x7f3de0195590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 0x7f3de0195590 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800b3d0 tx=0x7f3dd800b890 comp rx=0 tx=0).stop 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 shutdown_connections 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f3dcc06cc50 0x7f3dcc06f110 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3de00ffb90 0x7f3de0195ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 --2- 172.21.15.90:0/1470747124 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f3de00fe9a0 0x7f3de0195590 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.015+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 >> 172.21.15.90:0/1470747124 conn(0x7f3de00fa100 msgr2=0x7f3de0067fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.017 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.016+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 shutdown_connections 2024-09-13T15:58:48.018 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.016+0000 7f3de7133640 1 -- 172.21.15.90:0/1470747124 wait complete. 2024-09-13T15:58:48.028 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.foo.data pg_num 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 -- 172.21.15.90:0/90369892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 msgr2=0x7f08ec101100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 --2- 172.21.15.90:0/90369892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec101100 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f08dc009880 tx=0x7f08dc02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 -- 172.21.15.90:0/90369892 shutdown_connections 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 --2- 172.21.15.90:0/90369892 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec101100 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 --2- 172.21.15.90:0/90369892 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08ec0ffa90 0x7f08ec0ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 -- 172.21.15.90:0/90369892 >> 172.21.15.90:0/90369892 conn(0x7f08ec0fb1f0 msgr2=0x7f08ec0fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 -- 172.21.15.90:0/90369892 shutdown_connections 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.128+0000 7f08f468e640 1 -- 172.21.15.90:0/90369892 wait complete. 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f468e640 1 Processor -- start 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f468e640 1 -- start start 2024-09-13T15:58:48.131 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f468e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08ec0ffa90 0x7f08ec195590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f468e640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec195ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f468e640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec196010 con 0x7f08ec100c80 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f468e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec196180 con 0x7f08ec0ffa90 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec195ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec195ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36726/0 (socket says 172.21.15.90:36726) 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 -- 172.21.15.90:0/3448397104 learned_addr learned my addr 172.21.15.90:0/3448397104 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 -- 172.21.15.90:0/3448397104 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08ec0ffa90 msgr2=0x7f08ec195590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08ec0ffa90 0x7f08ec195590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 -- 172.21.15.90:0/3448397104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08dc009560 con 0x7f08ec100c80 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08f1c29640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec195ad0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f08dc005940 tx=0x7f08dc005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.129+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08dc0055d0 con 0x7f08ec100c80 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.130+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec19abf0 con 0x7f08ec100c80 2024-09-13T15:58:48.132 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.130+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f08dc003a80 con 0x7f08ec100c80 2024-09-13T15:58:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.130+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08dc03cab0 con 0x7f08ec100c80 2024-09-13T15:58:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.130+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec19b140 con 0x7f08ec100c80 2024-09-13T15:58:48.133 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.131+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec0fff10 con 0x7f08ec100c80 2024-09-13T15:58:48.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.131+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f08dc003bf0 con 0x7f08ec100c80 2024-09-13T15:58:48.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.132+0000 7f08e37fe640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f08d806cc50 0x7f08d806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.132+0000 7f08f242a640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f08d806cc50 0x7f08d806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.132+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f08dc0add90 con 0x7f08ec100c80 2024-09-13T15:58:48.134 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.132+0000 7f08f242a640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f08d806cc50 0x7f08d806f110 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f08ec100ae0 tx=0x7f08e8006c90 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.137 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.136+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f08dc077980 con 0x7f08ec100c80 2024-09-13T15:58:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.299+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"} v 0) v1 -- 0x7f08ec19b330 con 0x7f08ec100c80 2024-09-13T15:58:48.300 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.299+0000 7f08e37fe640 1 -- 172.21.15.90:0/3448397104 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f08dc07afa0 con 0x7f08ec100c80 2024-09-13T15:58:48.300 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f08d806cc50 msgr2=0x7f08d806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f08d806cc50 0x7f08d806f110 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f08ec100ae0 tx=0x7f08e8006c90 comp rx=0 tx=0).stop 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 msgr2=0x7f08ec195ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec195ad0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f08dc005940 tx=0x7f08dc005000 comp rx=0 tx=0).stop 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 shutdown_connections 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f08d806cc50 0x7f08d806f110 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.303 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f08ec100c80 0x7f08ec195ad0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 --2- 172.21.15.90:0/3448397104 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f08ec0ffa90 0x7f08ec195590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 >> 172.21.15.90:0/3448397104 conn(0x7f08ec0fb1f0 msgr2=0x7f08ec103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 shutdown_connections 2024-09-13T15:58:48.304 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.301+0000 7f08f468e640 1 -- 172.21.15.90:0/3448397104 wait complete. 2024-09-13T15:58:48.312 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:48 smithi090 ceph-mon[27811]: pgmap v1004: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 1 op/s 2024-09-13T15:58:48.312 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:48 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1211563821' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:58:48.313 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:48 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/681811062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:58:48.313 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:48 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3639477296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:58:48.313 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:48 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1470747124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:48.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.meta pg_num 2024-09-13T15:58:48.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 -- 172.21.15.90:0/3027095448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 msgr2=0x7f6e1c1028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 --2- 172.21.15.90:0/3027095448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1028e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009c20 tx=0x7f6e1402a970 comp rx=0 tx=0).stop 2024-09-13T15:58:48.418 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 -- 172.21.15.90:0/3027095448 shutdown_connections 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 --2- 172.21.15.90:0/3027095448 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e1c102e20 0x7f6e1c105210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 --2- 172.21.15.90:0/3027095448 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1028e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 -- 172.21.15.90:0/3027095448 >> 172.21.15.90:0/3027095448 conn(0x7f6e1c0fa0f0 msgr2=0x7f6e1c0fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 -- 172.21.15.90:0/3027095448 shutdown_connections 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.414+0000 7f6e21908640 1 -- 172.21.15.90:0/3027095448 wait complete. 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e21908640 1 Processor -- start 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e21908640 1 -- start start 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e21908640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e21908640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e1c102e20 0x7f6e1c195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e21908640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c195e60 con 0x7f6e1c1004f0 2024-09-13T15:58:48.419 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e21908640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c195fd0 con 0x7f6e1c102e20 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e1affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e1affd640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36748/0 (socket says 172.21.15.90:36748) 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.415+0000 7f6e1affd640 1 -- 172.21.15.90:0/3843883658 learned_addr learned my addr 172.21.15.90:0/3843883658 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e1a7fc640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e1c102e20 0x7f6e1c195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e1affd640 1 -- 172.21.15.90:0/3843883658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e1c102e20 msgr2=0x7f6e1c195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e1affd640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e1c102e20 0x7f6e1c195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e1affd640 1 -- 172.21.15.90:0/3843883658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e14009900 con 0x7f6e1c1004f0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e1affd640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1953e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009d40 tx=0x7f6e140308a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e14039620 con 0x7f6e1c1004f0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e14030ca0 con 0x7f6e1c1004f0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c19aa40 con 0x7f6e1c1004f0 2024-09-13T15:58:48.420 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e1403a970 con 0x7f6e1c1004f0 2024-09-13T15:58:48.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.416+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c19af10 con 0x7f6e1c1004f0 2024-09-13T15:58:48.421 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.417+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c1008d0 con 0x7f6e1c1004f0 2024-09-13T15:58:48.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.418+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f6e1400a040 con 0x7f6e1c1004f0 2024-09-13T15:58:48.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.418+0000 7f6e20906640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6e0806cc50 0x7f6e0806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.418+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f6e1402c040 con 0x7f6e1c1004f0 2024-09-13T15:58:48.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.418+0000 7f6e1a7fc640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6e0806cc50 0x7f6e0806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.422 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.419+0000 7f6e1a7fc640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6e0806cc50 0x7f6e0806f110 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e04005c70 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.423 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.422+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f6e14077b40 con 0x7f6e1c1004f0 2024-09-13T15:58:48.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:48 smithi175 ceph-mon[36128]: pgmap v1004: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 1 op/s 2024-09-13T15:58:48.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:48 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1211563821' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:58:48.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:48 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/681811062' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-13T15:58:48.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:48 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3639477296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-13T15:58:48.516 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:48 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1470747124' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:48.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.585+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"} v 0) v1 -- 0x7f6e1c19b3d0 con 0x7f6e1c1004f0 2024-09-13T15:58:48.587 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.585+0000 7f6e20906640 1 -- 172.21.15.90:0/3843883658 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6e1407b160 con 0x7f6e1c1004f0 2024-09-13T15:58:48.587 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:48.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.587+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6e0806cc50 msgr2=0x7f6e0806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.587+0000 7f6e21908640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6e0806cc50 0x7f6e0806f110 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6e04005fd0 tx=0x7f6e04005c70 comp rx=0 tx=0).stop 2024-09-13T15:58:48.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.587+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 msgr2=0x7f6e1c1953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.587+0000 7f6e21908640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1953e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6e14009d40 tx=0x7f6e140308a0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.590 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 shutdown_connections 2024-09-13T15:58:48.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f6e0806cc50 0x7f6e0806f110 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e1c102e20 0x7f6e1c195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 --2- 172.21.15.90:0/3843883658 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f6e1c1004f0 0x7f6e1c1953e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 >> 172.21.15.90:0/3843883658 conn(0x7f6e1c0fa0f0 msgr2=0x7f6e1c0fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 shutdown_connections 2024-09-13T15:58:48.591 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.588+0000 7f6e21908640 1 -- 172.21.15.90:0/3843883658 wait complete. 2024-09-13T15:58:48.600 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.bar.data pg_num 2024-09-13T15:58:48.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 -- 172.21.15.90:0/4157778481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58102ec0 msgr2=0x7efc581052b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 --2- 172.21.15.90:0/4157778481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58102ec0 0x7efc581052b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efc4c009880 tx=0x7efc4c02a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 -- 172.21.15.90:0/4157778481 shutdown_connections 2024-09-13T15:58:48.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 --2- 172.21.15.90:0/4157778481 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58102ec0 0x7efc581052b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.702 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 --2- 172.21.15.90:0/4157778481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc58100590 0x7efc58102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 -- 172.21.15.90:0/4157778481 >> 172.21.15.90:0/4157778481 conn(0x7efc580fa0f0 msgr2=0x7efc580fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 -- 172.21.15.90:0/4157778481 shutdown_connections 2024-09-13T15:58:48.703 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 -- 172.21.15.90:0/4157778481 wait complete. 2024-09-13T15:58:48.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.701+0000 7efc5cb26640 1 Processor -- start 2024-09-13T15:58:48.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 -- start start 2024-09-13T15:58:48.704 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 0x7efc58195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc58102ec0 0x7efc581959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58195ee0 con 0x7efc58100590 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58196050 con 0x7efc58102ec0 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 0x7efc58195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 0x7efc58195460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36772/0 (socket says 172.21.15.90:36772) 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 -- 172.21.15.90:0/3713341869 learned_addr learned my addr 172.21.15.90:0/3713341869 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 -- 172.21.15.90:0/3713341869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc58102ec0 msgr2=0x7efc581959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc58102ec0 0x7efc581959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 -- 172.21.15.90:0/3713341869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c009560 con 0x7efc58100590 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5659c640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 0x7efc58195460 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efc4800d6f0 tx=0x7efc4800dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc48009a30 con 0x7efc58100590 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5819ab20 con 0x7efc58100590 2024-09-13T15:58:48.705 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efc480054b0 con 0x7efc58100590 2024-09-13T15:58:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc48004d00 con 0x7efc58100590 2024-09-13T15:58:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.702+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5819b0d0 con 0x7efc58100590 2024-09-13T15:58:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.703+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc58100970 con 0x7efc58100590 2024-09-13T15:58:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.704+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7efc48005020 con 0x7efc58100590 2024-09-13T15:58:48.706 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.705+0000 7efc477fe640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7efc4006cc50 0x7efc4006f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.705+0000 7efc55d9b640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7efc4006cc50 0x7efc4006f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.705+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7efc4808c570 con 0x7efc58100590 2024-09-13T15:58:48.707 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.705+0000 7efc55d9b640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7efc4006cc50 0x7efc4006f110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efc4c02a9c0 tx=0x7efc4c000bc0 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.710 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.709+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7efc48056160 con 0x7efc58100590 2024-09-13T15:58:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.871+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"} v 0) v1 -- 0x7efc5819b2c0 con 0x7efc58100590 2024-09-13T15:58:48.873 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.872+0000 7efc477fe640 1 -- 172.21.15.90:0/3713341869 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]=0 v72) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc48059780 con 0x7efc58100590 2024-09-13T15:58:48.873 INFO:teuthology.orchestra.run.smithi090.stdout:pg_num: 32 2024-09-13T15:58:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7efc4006cc50 msgr2=0x7efc4006f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7efc4006cc50 0x7efc4006f110 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efc4c02a9c0 tx=0x7efc4c000bc0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 msgr2=0x7efc58195460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 0x7efc58195460 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efc4800d6f0 tx=0x7efc4800dbb0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 shutdown_connections 2024-09-13T15:58:48.875 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7efc4006cc50 0x7efc4006f110 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc58102ec0 0x7efc581959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 --2- 172.21.15.90:0/3713341869 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7efc58100590 0x7efc58195460 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 >> 172.21.15.90:0/3713341869 conn(0x7efc580fa0f0 msgr2=0x7efc580fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 shutdown_connections 2024-09-13T15:58:48.876 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.874+0000 7efc5cb26640 1 -- 172.21.15.90:0/3713341869 wait complete. 2024-09-13T15:58:48.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T15:58:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 -- 172.21.15.90:0/1100132795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 msgr2=0x7f05c00feda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1100132795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c00feda0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009880 tx=0x7f05b002a4a0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 -- 172.21.15.90:0/1100132795 shutdown_connections 2024-09-13T15:58:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1100132795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 0x7f05c0100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.988 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1100132795 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c00feda0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 -- 172.21.15.90:0/1100132795 >> 172.21.15.90:0/1100132795 conn(0x7f05c00fa100 msgr2=0x7f05c00fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 -- 172.21.15.90:0/1100132795 shutdown_connections 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.985+0000 7f05c5d76640 1 -- 172.21.15.90:0/1100132795 wait complete. 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05c5d76640 1 Processor -- start 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05c5d76640 1 -- start start 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05c5d76640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c0194400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05c5d76640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 0x7f05c0194940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05c5d76640 1 -- --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0194e80 con 0x7f05c00fe9a0 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05c5d76640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c0194ff0 con 0x7f05c00ffb90 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c0194400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05bf7fe640 1 --2- >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c0194400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.90:3300/0 says I am v2:172.21.15.90:36800/0 (socket says 172.21.15.90:36800) 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05bf7fe640 1 -- 172.21.15.90:0/1003399322 learned_addr learned my addr 172.21.15.90:0/1003399322 (peer_addr_for_me v2:172.21.15.90:0/0) 2024-09-13T15:58:48.989 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05beffd640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 0x7f05c0194940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05bf7fe640 1 -- 172.21.15.90:0/1003399322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 msgr2=0x7f05c0194940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05bf7fe640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 0x7f05c0194940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05bf7fe640 1 -- 172.21.15.90:0/1003399322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0009560 con 0x7f05c00fe9a0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.986+0000 7f05beffd640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 0x7f05c0194940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.987+0000 7f05bf7fe640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c0194400 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f05b0006010 tx=0x7f05b0005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.987+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05b0005600 con 0x7f05c00fe9a0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.987+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f05b0003a80 con 0x7f05c00fe9a0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.987+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05b003ca90 con 0x7f05c00fe9a0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.987+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c0199a60 con 0x7f05c00fe9a0 2024-09-13T15:58:48.990 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.987+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c0199fb0 con 0x7f05c00fe9a0 2024-09-13T15:58:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.988+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c0107430 con 0x7f05c00fe9a0 2024-09-13T15:58:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.988+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 4 ==== mgrmap(e 19) v1 ==== 90850+0+0 (secure 0 0 0) 0x7f05b0003bf0 con 0x7f05c00fe9a0 2024-09-13T15:58:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.989+0000 7f05bcff9640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f05a806cc50 0x7f05a806f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T15:58:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.989+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7340+0+0 (secure 0 0 0) 0x7f05b00add50 con 0x7f05c00fe9a0 2024-09-13T15:58:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.989+0000 7f05beffd640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f05a806cc50 0x7f05a806f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T15:58:48.991 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.989+0000 7f05beffd640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f05a806cc50 0x7f05a806f110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f05b4006fd0 tx=0x7f05b4008040 comp rx=0 tx=0).ready entity=mgr.14217 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T15:58:48.994 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:48.993+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183819 (secure 0 0 0) 0x7f05b0077940 con 0x7f05c00fe9a0 2024-09-13T15:58:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.167+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 --> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f05c0107640 con 0x7f05c00fe9a0 2024-09-13T15:58:49.169 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.168+0000 7f05bcff9640 1 -- 172.21.15.90:0/1003399322 <== mon.0 v2:172.21.15.90:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v72) v1 ==== 81+0+1010 (secure 0 0 0) 0x7f05b007af60 con 0x7f05c00fe9a0 2024-09-13T15:58:49.169 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6832/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.175:6833/2042693132 2024-09-14T15:33:06.895655+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3168219877 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3860788333 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/575512074 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/1322912193 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6800/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:49.173 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2223577157 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/782157526 2024-09-14T15:25:26.727266+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/1586604027 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3565908774 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/3782750394 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/2606530289 2024-09-14T15:26:07.051031+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:0/4162862940 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:49.174 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90:6801/3319538530 2024-09-14T15:27:18.494403+0000 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:listed 17 entries 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f05a806cc50 msgr2=0x7f05a806f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f05a806cc50 0x7f05a806f110 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f05b4006fd0 tx=0x7f05b4008040 comp rx=0 tx=0).stop 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 msgr2=0x7f05c0194400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c0194400 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f05b0006010 tx=0x7f05b0005000 comp rx=0 tx=0).stop 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 shutdown_connections 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:6800/885641711,v1:172.21.15.90:6801/885641711] conn(0x7f05a806cc50 0x7f05a806f110 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:49.175 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f05c00ffb90 0x7f05c0194940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 --2- 172.21.15.90:0/1003399322 >> [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] conn(0x7f05c00fe9a0 0x7f05c0194400 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T15:58:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 >> 172.21.15.90:0/1003399322 conn(0x7f05c00fa100 msgr2=0x7f05c0102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T15:58:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 shutdown_connections 2024-09-13T15:58:49.176 INFO:teuthology.orchestra.run.smithi090.stderr:2024-09-13T15:58:49.170+0000 7f05c5d76640 1 -- 172.21.15.90:0/1003399322 wait complete. 2024-09-13T15:58:49.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:49 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3448397104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:49.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:49 smithi090 ceph-mon[27811]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:49.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:49 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3843883658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:49.181 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:49 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/3713341869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:49.182 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi175.front.sepia.ceph.com... 2024-09-13T15:58:49.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:49.182 DEBUG:teuthology.orchestra.run.smithi175:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-13T15:58:49.240 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T15:58:49.241 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[65134]: fuse finished with error 0 and tester_r 0 2024-09-13T15:58:49.409 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:49 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3448397104' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.foo.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:49.410 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:49 smithi175 ceph-mon[36128]: from='mgr.14217 172.21.15.90:0/2926107581' entity='mgr.smithi090.fyvvpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-13T15:58:49.410 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:49 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3843883658' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.meta", "var": "pg_num"}]: dispatch 2024-09-13T15:58:49.410 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:49 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/3713341869' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.bar.data", "var": "pg_num"}]: dispatch 2024-09-13T15:58:50.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:50 smithi090 ceph-mon[27811]: pgmap v1005: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 0 op/s 2024-09-13T15:58:50.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:50 smithi090 ceph-mon[27811]: from='client.? 172.21.15.90:0/1003399322' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:58:50.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:50 smithi175 ceph-mon[36128]: pgmap v1005: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 170 B/s rd, 938 B/s wr, 0 op/s 2024-09-13T15:58:50.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:50 smithi175 ceph-mon[36128]: from='client.? 172.21.15.90:0/1003399322' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-13T15:58:52.433 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:52 smithi175 ceph-mon[36128]: pgmap v1006: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:58:52.501 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:52 smithi090 ceph-mon[27811]: pgmap v1006: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-13T15:58:54.504 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:54 smithi090 ceph-mon[27811]: pgmap v1007: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:58:54.515 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:54 smithi175 ceph-mon[36128]: pgmap v1007: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:58:55.343 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-09-13T15:58:55.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.343 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T15:58:55.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.381 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-09-13T15:58:55.439 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T15:58:55.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.440 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T15:58:55.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.510 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link delete ceph-brx 2024-09-13T15:58:55.617 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-13T15:58:55.627 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-13T15:58:55.627 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2024-09-13T15:58:55.627 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T15:58:55.667 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056009,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497436k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1157 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248716k,nr_inodes=812179,mode=700,uid=1000,gid=1333,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/d2030a90e3b5fab53deea5dd89f324d41c99be798780254d7fe66216636313e5/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/d2030a90e3b5fab53deea5dd89f324d41c99be798780254d7fe66216636313e5/diff,workdir=/var/lib/containers/storage/overlay/d2030a90e3b5fab53deea5dd89f324d41c99be798780254d7fe66216636313e5/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d2acd8730312a617d1e0ab2f75a6789573e1ad633b9419fef7feaaa7ba72c7da/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/d2acd8730312a617d1e0ab2f75a6789573e1ad633b9419fef7feaaa7ba72c7da/diff,workdir=/var/lib/containers/storage/overlay/d2acd8730312a617d1e0ab2f75a6789573e1ad633b9419fef7feaaa7ba72c7da/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248716k,nr_inodes=812179,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/2d44ee523a8e752dfe79235ca13286dfd11746ac7ec06ca7e83d4609a3b2ff3a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/2d44ee523a8e752dfe79235ca13286dfd11746ac7ec06ca7e83d4609a3b2ff3a/diff,workdir=/var/lib/containers/storage/overlay/2d44ee523a8e752dfe79235ca13286dfd11746ac7ec06ca7e83d4609a3b2ff3a/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/e362becb892e441831077d95d56cdacd9eeb1a5a1b330f88678b3a89c2944dd1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/e362becb892e441831077d95d56cdacd9eeb1a5a1b330f88678b3a89c2944dd1/diff,workdir=/var/lib/containers/storage/overlay/e362becb892e441831077d95d56cdacd9eeb1a5a1b330f88678b3a89c2944dd1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2c0da5e373c64d80fb4752499c01d454f78d30d593402e23d443892df002e76e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/3HQWYQKKDBVQBIOYYAIB7GUESU:/var/lib/containers/storage/overlay/l/DTUXZHTHT366KVCOMIG7TM7MAW:/var/lib/containers/storage/overlay/l/ZFWACTLJFMSXWDIU5PQBIGLQYO,upperdir=/var/lib/containers/storage/overlay/2c0da5e373c64d80fb4752499c01d454f78d30d593402e23d443892df002e76e/diff,workdir=/var/lib/containers/storage/overlay/2c0da5e373c64d80fb4752499c01d454f78d30d593402e23d443892df002e76e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d5113077bad9da3a2ecca766855d2f238dfe96e5877076687267e223f804bcc0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Z4EMWW5NSKIPEG52UE5YMI7LJF:/var/lib/containers/storage/overlay/l/LMCJ6C4VY2VKHNDLG7S3IQYMNQ:/var/lib/containers/storage/overlay/l/3JQTE3AP3KSLT3WF6OVEXEKLWA:/var/lib/containers/storage/overlay/l/G6W5HO3F4XWFNVMBBJQLMQUGLA:/var/lib/containers/storage/overlay/l/3Y4OVLBUDPSF7YG4WJEI7XMDSP:/var/lib/containers/storage/overlay/l/2DINXGRMU56OTTAYYK75WKCHDO,upperdir=/var/lib/containers/storage/overlay/d5113077bad9da3a2ecca766855d2f238dfe96e5877076687267e223f804bcc0/diff,workdir=/var/lib/containers/storage/overlay/d5113077bad9da3a2ecca766855d2f238dfe96e5877076687267e223f804bcc0/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/63980f086759bf5f93166998ad95b95d6ff48b9831b1a5440dda958d9b50d023/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/X7PQPVB6QYFDKPYKO3KS7G54PQ,upperdir=/var/lib/containers/storage/overlay/63980f086759bf5f93166998ad95b95d6ff48b9831b1a5440dda958d9b50d023/diff,workdir=/var/lib/containers/storage/overlay/63980f086759bf5f93166998ad95b95d6ff48b9831b1a5440dda958d9b50d023/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/186eafb078d00128fa345c6d7042a2b3deead86d829b44849b0a63c68fd399f1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/JQD6WPILTZNZMS5OFTPJI26RYK:/var/lib/containers/storage/overlay/l/TPC3OUD6S63SGNFSISNMQ27PXI:/var/lib/containers/storage/overlay/l/E6BBXI6AD6P6BRC4PBOZOBR7PT:/var/lib/containers/storage/overlay/l/4UAJVSRVJX66IFQEJUF7PY57G7:/var/lib/containers/storage/overlay/l/E35MLFJFW3HJOGPXMO4G7NWQAU:/var/lib/containers/storage/overlay/l/3BKYZVOKHHB6YX7YFL46JBUP4C:/var/lib/containers/storage/overlay/l/LEPBDKLOQGKU466HFYPGTJ3X7F:/var/lib/containers/storage/overlay/l/AUWNUX3CPSEXPPRKAEQDF4DMLO:/var/lib/containers/storage/overlay/l/TKYDECXUKIFLWSFUZYPRCVDKXE:/var/lib/containers/storage/overlay/l/PMWVGD7YD62Z33VLCMKJSVE7DJ:/var/lib/containers/storage/overlay/l/DRZCDO3DAKMK6TLTNNF2V6MRMN:/var/lib/containers/storage/overlay/l/BASSEHRKL7C5IAFANV3D7GUANB,upperdir=/var/lib/containers/storage/overlay/186eafb078d00128fa345c6d7042a2b3deead86d829b44849b0a63c68fd399f1/diff,workdir=/var/lib/containers/storage/overlay/186eafb078d00128fa345c6d7042a2b3deead86d829b44849b0a63c68fd399f1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/cafcb4817a8454dcb5e1c19aacc2c88f314b13afc8751ac7cdb04acdbf398b24/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/cafcb4817a8454dcb5e1c19aacc2c88f314b13afc8751ac7cdb04acdbf398b24/diff,workdir=/var/lib/containers/storage/overlay/cafcb4817a8454dcb5e1c19aacc2c88f314b13afc8751ac7cdb04acdbf398b24/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/bc236cb800e5995e644095aeba59887ecfb7c2b21ac72d19b3691a3c76718140/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/bc236cb800e5995e644095aeba59887ecfb7c2b21ac72d19b3691a3c76718140/diff,workdir=/var/lib/containers/storage/overlay/bc236cb800e5995e644095aeba59887ecfb7c2b21ac72d19b3691a3c76718140/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/13cd35e2d0c101ff2491eadc4aa455b4bcef61f83f0cf79f2c00a3c143f85d4f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/13cd35e2d0c101ff2491eadc4aa455b4bcef61f83f0cf79f2c00a3c143f85d4f/diff,workdir=/var/lib/containers/storage/overlay/13cd35e2d0c101ff2491eadc4aa455b4bcef61f83f0cf79f2c00a3c143f85d4f/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/5910410d1306892b310ae6b5023ae6127e93d95829908dcd6498e0aca7c7847e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/5910410d1306892b310ae6b5023ae6127e93d95829908dcd6498e0aca7c7847e/diff,workdir=/var/lib/containers/storage/overlay/5910410d1306892b310ae6b5023ae6127e93d95829908dcd6498e0aca7c7847e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/47077534a3746e55f66fa8bb1c0bfb72bb9acb34da48fc44456cad8df55f86ee/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/47077534a3746e55f66fa8bb1c0bfb72bb9acb34da48fc44456cad8df55f86ee/diff,workdir=/var/lib/containers/storage/overlay/47077534a3746e55f66fa8bb1c0bfb72bb9acb34da48fc44456cad8df55f86ee/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9494e6bab7c5f83d65ae24f30f7d32d2e5f81bd3ca6f0203cda1d13509dd0837/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/PGWX3JNEGDEA4XSB5HOHQGFOYD:/var/lib/containers/storage/overlay/l/3AXV4XN62FU7VYDH775OQJIVWG,upperdir=/var/lib/containers/storage/overlay/9494e6bab7c5f83d65ae24f30f7d32d2e5f81bd3ca6f0203cda1d13509dd0837/diff,workdir=/var/lib/containers/storage/overlay/9494e6bab7c5f83d65ae24f30f7d32d2e5f81bd3ca6f0203cda1d13509dd0837/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497436k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-09-13T15:58:55.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.667 DEBUG:teuthology.orchestra.run.smithi090:> ip netns list 2024-09-13T15:58:55.728 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T15:58:55.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T15:58:55.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T15:58:55.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo ip link delete ceph-brx 2024-09-13T15:58:55.889 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-13T15:58:55.899 INFO:tasks.cephadm:Teardown begin 2024-09-13T15:58:55.899 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:58:55.955 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T15:58:56.003 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-13T15:58:56.003 DEBUG:teuthology.orchestra.run.smithi090:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:58:56.019 DEBUG:teuthology.orchestra.run.smithi175:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-13T15:58:56.035 INFO:tasks.cephadm:Stopping all daemons... 2024-09-13T15:58:56.036 INFO:tasks.cephadm.mon.smithi090:Stopping mon.smithi090... 2024-09-13T15:58:56.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090 2024-09-13T15:58:56.367 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 ceph-mon[27811]: pgmap v1008: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:58:56.368 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 systemd[1]: Stopping Ceph mon.smithi090 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:58:56.505 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:56 smithi175 ceph-mon[36128]: pgmap v1008: 129 pgs: 129 active+clean; 226 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-13T15:58:56.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090[27807]: 2024-09-13T15:58:56.366+0000 7f24c3a6e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi090 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:58:56.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090[27807]: 2024-09-13T15:58:56.366+0000 7f24c3a6e640 -1 mon.smithi090@0(leader) e2 *** Got Signal Terminated *** 2024-09-13T15:58:56.672 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 podman[87356]: 2024-09-13 15:58:56.397205183 +0000 UTC m=+0.149564268 container died 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-13T15:58:57.001 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 podman[87356]: 2024-09-13 15:58:56.876585728 +0000 UTC m=+0.628944842 container cleanup 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux ) 2024-09-13T15:58:57.002 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:56 smithi090 bash[87356]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090 2024-09-13T15:58:57.309 INFO:journalctl@ceph.mon.smithi090.smithi090.stdout:Sep 13 15:58:57 smithi090 podman[87369]: 2024-09-13 15:58:57.196810879 +0000 UTC m=+0.795635136 container remove 6e8df88e6303e90daacf536462a4a282f58601787c27198d17a7211e3b3fbf1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:50bbe7f395997dcd871813949c49addbbf630361, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi090, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-13T15:58:57.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi090.service' 2024-09-13T15:58:57.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:58:57.395 INFO:tasks.cephadm.mon.smithi090:Stopped mon.smithi090 2024-09-13T15:58:57.395 INFO:tasks.cephadm.mon.smithi175:Stopping mon.smithi175... 2024-09-13T15:58:57.396 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi175 2024-09-13T15:58:57.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:57 smithi175 systemd[1]: Stopping Ceph mon.smithi175 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:58:58.265 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:57 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi175[36057]: 2024-09-13T15:58:57.987+0000 7ff7a9ad6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi175 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:58:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:57 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi175[36057]: 2024-09-13T15:58:57.987+0000 7ff7a9ad6640 -1 mon.smithi175@1(peon) e2 *** Got Signal Terminated *** 2024-09-13T15:58:58.266 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:58 smithi175 podman[72503]: 2024-09-13 15:58:58.071289403 +0000 UTC m=+0.216232247 container died b1ecf9d5db057cc010e54248293818c39cd70caade7ab99928d15c3fe12f09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi175, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:58:58.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:58 smithi175 podman[72503]: 2024-09-13 15:58:58.573024586 +0000 UTC m=+0.717967429 container cleanup b1ecf9d5db057cc010e54248293818c39cd70caade7ab99928d15c3fe12f09a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi175, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-13T15:58:58.765 INFO:journalctl@ceph.mon.smithi175.smithi175.stdout:Sep 13 15:58:58 smithi175 bash[72503]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-mon-smithi175 2024-09-13T15:58:59.010 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@mon.smithi175.service' 2024-09-13T15:58:59.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:58:59.061 INFO:tasks.cephadm.mon.smithi175:Stopped mon.smithi175 2024-09-13T15:58:59.061 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-13T15:58:59.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.0 2024-09-13T15:58:59.348 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:58:59 smithi090 systemd[1]: Stopping Ceph osd.0 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:58:59.751 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:58:59 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0[45757]: 2024-09-13T15:58:59.417+0000 7fb73c487640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:58:59.751 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:58:59 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0[45757]: 2024-09-13T15:58:59.417+0000 7fb73c487640 -1 osd.0 72 *** Got signal Terminated *** 2024-09-13T15:58:59.751 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:58:59 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0[45757]: 2024-09-13T15:58:59.417+0000 7fb73c487640 -1 osd.0 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:05.209 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:04 smithi090 podman[87502]: 2024-09-13 15:59:04.897082743 +0000 UTC m=+5.637784218 container died 53857c694096aa99565d030094d338bb55f7def7e635cdceb3ac4b069ed42e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-13T15:59:05.999 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:05 smithi090 podman[87502]: 2024-09-13 15:59:05.703110683 +0000 UTC m=+6.443812156 container cleanup 53857c694096aa99565d030094d338bb55f7def7e635cdceb3ac4b069ed42e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:59:05.999 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:05 smithi090 bash[87502]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0 2024-09-13T15:59:06.252 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:05 smithi090 podman[87520]: 2024-09-13 15:59:05.997751804 +0000 UTC m=+1.087540510 container remove 53857c694096aa99565d030094d338bb55f7def7e635cdceb3ac4b069ed42e52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-13T15:59:06.611 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:06 smithi090 podman[87609]: 2024-09-13 15:59:06.213762486 +0000 UTC m=+0.016570124 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:06.612 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:06 smithi090 podman[87609]: 2024-09-13 15:59:06.340942812 +0000 UTC m=+0.143750451 container create 93fd6c2a7a9b05f80e6c86f6e90a3da8b612cab66420de7e4ccbfa606a15ec5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-13T15:59:06.612 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:06 smithi090 podman[87609]: 2024-09-13 15:59:06.61116979 +0000 UTC m=+0.413977429 container init 93fd6c2a7a9b05f80e6c86f6e90a3da8b612cab66420de7e4ccbfa606a15ec5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:59:07.001 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:06 smithi090 podman[87609]: 2024-09-13 15:59:06.614687199 +0000 UTC m=+0.417494847 container start 93fd6c2a7a9b05f80e6c86f6e90a3da8b612cab66420de7e4ccbfa606a15ec5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-13T15:59:07.001 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:06 smithi090 podman[87609]: 2024-09-13 15:59:06.727964004 +0000 UTC m=+0.530771646 container attach 93fd6c2a7a9b05f80e6c86f6e90a3da8b612cab66420de7e4ccbfa606a15ec5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=) 2024-09-13T15:59:07.324 INFO:journalctl@ceph.osd.0.smithi090.stdout:Sep 13 15:59:07 smithi090 podman[87609]: 2024-09-13 15:59:07.019852775 +0000 UTC m=+0.822660415 container died 93fd6c2a7a9b05f80e6c86f6e90a3da8b612cab66420de7e4ccbfa606a15ec5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-13T15:59:07.619 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.0.service' 2024-09-13T15:59:07.659 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:07.659 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-13T15:59:07.659 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-13T15:59:07.659 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.1 2024-09-13T15:59:08.001 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:07 smithi090 systemd[1]: Stopping Ceph osd.1 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:08.501 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:08 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1[52224]: 2024-09-13T15:59:08.020+0000 7f8eed8d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:08.501 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:08 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1[52224]: 2024-09-13T15:59:08.020+0000 7f8eed8d9640 -1 osd.1 72 *** Got signal Terminated *** 2024-09-13T15:59:08.501 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:08 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1[52224]: 2024-09-13T15:59:08.020+0000 7f8eed8d9640 -1 osd.1 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:13.874 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:13 smithi090 podman[87713]: 2024-09-13 15:59:13.530972727 +0000 UTC m=+5.645325455 container died a9705caa0b6d44c6619bda704b02e34576fc7c9d63ea9dce2abc506e0af63074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:14.184 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:13 smithi090 podman[87713]: 2024-09-13 15:59:13.965159839 +0000 UTC m=+6.079512557 container cleanup a9705caa0b6d44c6619bda704b02e34576fc7c9d63ea9dce2abc506e0af63074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:59:14.184 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:13 smithi090 bash[87713]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1 2024-09-13T15:59:14.501 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:14 smithi090 podman[87732]: 2024-09-13 15:59:14.183713161 +0000 UTC m=+0.621123371 container remove a9705caa0b6d44c6619bda704b02e34576fc7c9d63ea9dce2abc506e0af63074 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-13T15:59:14.839 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:14 smithi090 podman[87816]: 2024-09-13 15:59:14.414955384 +0000 UTC m=+0.015774863 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:14.839 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:14 smithi090 podman[87816]: 2024-09-13 15:59:14.576906476 +0000 UTC m=+0.177725954 container create 6a45cb9cea52445e83c20cfdfbc19dba5711df51175398b8e70eb52ab14bebcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:14.839 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:14 smithi090 podman[87816]: 2024-09-13 15:59:14.838249298 +0000 UTC m=+0.439068776 container init 6a45cb9cea52445e83c20cfdfbc19dba5711df51175398b8e70eb52ab14bebcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-13T15:59:15.251 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:14 smithi090 podman[87816]: 2024-09-13 15:59:14.842159687 +0000 UTC m=+0.442979167 container start 6a45cb9cea52445e83c20cfdfbc19dba5711df51175398b8e70eb52ab14bebcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-13T15:59:15.251 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:14 smithi090 podman[87816]: 2024-09-13 15:59:14.938406572 +0000 UTC m=+0.539226065 container attach 6a45cb9cea52445e83c20cfdfbc19dba5711df51175398b8e70eb52ab14bebcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T15:59:15.251 INFO:journalctl@ceph.osd.1.smithi090.stdout:Sep 13 15:59:15 smithi090 podman[87816]: 2024-09-13 15:59:15.028241605 +0000 UTC m=+0.629061088 container died 6a45cb9cea52445e83c20cfdfbc19dba5711df51175398b8e70eb52ab14bebcf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-13T15:59:15.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.1.service' 2024-09-13T15:59:15.674 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:15.674 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-13T15:59:15.674 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-13T15:59:15.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.2 2024-09-13T15:59:16.004 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:15 smithi090 systemd[1]: Stopping Ceph osd.2 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:16.751 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:16 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2[58886]: 2024-09-13T15:59:16.281+0000 7fa616cf3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:16.751 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:16 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2[58886]: 2024-09-13T15:59:16.281+0000 7fa616cf3640 -1 osd.2 72 *** Got signal Terminated *** 2024-09-13T15:59:16.751 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:16 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2[58886]: 2024-09-13T15:59:16.281+0000 7fa616cf3640 -1 osd.2 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:21.960 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:21 smithi090 podman[87921]: 2024-09-13 15:59:21.640491477 +0000 UTC m=+5.742972067 container died 1c8532c84a7c0d973e78c3ba14e0452f3208fbb463d794e49f374e35b9a78a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.vendor=CentOS) 2024-09-13T15:59:22.252 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 podman[87921]: 2024-09-13 15:59:22.050630879 +0000 UTC m=+6.153111465 container cleanup 1c8532c84a7c0d973e78c3ba14e0452f3208fbb463d794e49f374e35b9a78a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:22.252 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 bash[87921]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2 2024-09-13T15:59:22.694 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 podman[87935]: 2024-09-13 15:59:22.285658159 +0000 UTC m=+0.645161059 container remove 1c8532c84a7c0d973e78c3ba14e0452f3208fbb463d794e49f374e35b9a78a71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-13T15:59:22.957 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 podman[88027]: 2024-09-13 15:59:22.596839861 +0000 UTC m=+0.016400767 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:22.958 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 podman[88027]: 2024-09-13 15:59:22.695155702 +0000 UTC m=+0.114716662 container create a13900e007a34d3e50c8cc28f76838ecac5db07beda763cd5c8a09c38ac265e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-13T15:59:22.958 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 podman[88027]: 2024-09-13 15:59:22.956870099 +0000 UTC m=+0.376430997 container init a13900e007a34d3e50c8cc28f76838ecac5db07beda763cd5c8a09c38ac265e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:23.251 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:22 smithi090 podman[88027]: 2024-09-13 15:59:22.96072871 +0000 UTC m=+0.380289619 container start a13900e007a34d3e50c8cc28f76838ecac5db07beda763cd5c8a09c38ac265e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T15:59:23.251 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:23 smithi090 podman[88027]: 2024-09-13 15:59:23.073721008 +0000 UTC m=+0.493281909 container attach a13900e007a34d3e50c8cc28f76838ecac5db07beda763cd5c8a09c38ac265e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:59:23.252 INFO:journalctl@ceph.osd.2.smithi090.stdout:Sep 13 15:59:23 smithi090 podman[88027]: 2024-09-13 15:59:23.148629972 +0000 UTC m=+0.568190880 container died a13900e007a34d3e50c8cc28f76838ecac5db07beda763cd5c8a09c38ac265e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:23.760 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.2.service' 2024-09-13T15:59:23.800 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:23.801 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-13T15:59:23.801 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-13T15:59:23.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.3 2024-09-13T15:59:24.001 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:23 smithi090 systemd[1]: Stopping Ceph osd.3 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:24.501 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:24 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3[66054]: 2024-09-13T15:59:24.140+0000 7f9ee0585640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:24.501 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:24 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3[66054]: 2024-09-13T15:59:24.140+0000 7f9ee0585640 -1 osd.3 72 *** Got signal Terminated *** 2024-09-13T15:59:24.501 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:24 smithi090 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3[66054]: 2024-09-13T15:59:24.140+0000 7f9ee0585640 -1 osd.3 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:25.515 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:25 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:25.261+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:26.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:26 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:26.290+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:26.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:26 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:26.280+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:27.265 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:27 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:27.149+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:27.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:27 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:27.334+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:27.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:27 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:27.306+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:28.515 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:28 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:28.350+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:28.516 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:28 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:28.174+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:28.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:28 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:28.333+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:28.516 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:28 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:28.168+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:29.515 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:29 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:29.341+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:29.516 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:29 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:29.129+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:29.516 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:29 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:29.210+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:29.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:29 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:29.370+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:29.949 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:29 smithi090 podman[88130]: 2024-09-13 15:59:29.665596939 +0000 UTC m=+5.635281187 container died 0a3867cad7fce0b56f25d8f740982c2f45bd57af5e69407e7c3888eeeee5aeea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:59:30.251 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 podman[88130]: 2024-09-13 15:59:30.025423377 +0000 UTC m=+5.995107623 container cleanup 0a3867cad7fce0b56f25d8f740982c2f45bd57af5e69407e7c3888eeeee5aeea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:30.251 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 bash[88130]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3 2024-09-13T15:59:30.390 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:30 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:30.086+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:30.391 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:30 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:30.201+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:30.391 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:30 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:30.327+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:30.688 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 podman[88159]: 2024-09-13 15:59:30.270032779 +0000 UTC m=+0.581908139 container remove 0a3867cad7fce0b56f25d8f740982c2f45bd57af5e69407e7c3888eeeee5aeea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:59:30.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:30 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:30.389+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:31.001 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 podman[88249]: 2024-09-13 15:59:30.687992024 +0000 UTC m=+0.110136105 container create 357d1f78a541c42950d89d0cabb1271b756a9f69e67266480bb9b1e14f939b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T15:59:31.001 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 podman[88249]: 2024-09-13 15:59:30.593924655 +0000 UTC m=+0.016068703 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:31.001 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 podman[88249]: 2024-09-13 15:59:30.932519295 +0000 UTC m=+0.354663336 container init 357d1f78a541c42950d89d0cabb1271b756a9f69e67266480bb9b1e14f939b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-13T15:59:31.001 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:30 smithi090 podman[88249]: 2024-09-13 15:59:30.935747155 +0000 UTC m=+0.357891197 container start 357d1f78a541c42950d89d0cabb1271b756a9f69e67266480bb9b1e14f939b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-13T15:59:31.417 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:31 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:31.065+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:31.418 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:31 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:31.216+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:31.418 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:31 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:31.286+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:31.422 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:31 smithi090 podman[88249]: 2024-09-13 15:59:31.040041276 +0000 UTC m=+0.462185325 container attach 357d1f78a541c42950d89d0cabb1271b756a9f69e67266480bb9b1e14f939b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T15:59:31.422 INFO:journalctl@ceph.osd.3.smithi090.stdout:Sep 13 15:59:31 smithi090 podman[88249]: 2024-09-13 15:59:31.124809925 +0000 UTC m=+0.546953973 container died 357d1f78a541c42950d89d0cabb1271b756a9f69e67266480bb9b1e14f939b9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-13T15:59:31.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.3.service' 2024-09-13T15:59:31.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:31 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:31.416+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:31.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:31.798 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-13T15:59:31.798 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-13T15:59:31.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.4 2024-09-13T15:59:32.188 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:31 smithi175 systemd[1]: Stopping Ceph osd.4 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:32.189 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:32.035+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:32.459 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:32.328+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:32.459 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:32.187+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:32.459 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:32.220+0000 7f1322594640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:32.459 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:32.220+0000 7f1322594640 -1 osd.4 72 *** Got signal Terminated *** 2024-09-13T15:59:32.459 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:32.220+0000 7f1322594640 -1 osd.4 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:32.459 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:32.457+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:33.265 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:32 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:32.986+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:33.266 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:33 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:33.217+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:33.555 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:33 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:33.418+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:33.555 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:33 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:33.281+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:34.324 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:34 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:34.198+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:34.324 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:34 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:34.032+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:34.324 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:34 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:34.032+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:34.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:34 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:34.433+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:34.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:34 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:34.323+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:35.368 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:35.068+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:35.369 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:35.068+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:35.369 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:35.229+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:35.369 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:35.229+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:35.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:35.428+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:35.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:35.367+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:35.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:35 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:35.367+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:36.458 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:36.246+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:36.458 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:36.246+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:36.459 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:36.327+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:36.459 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:36.327+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:36.459 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:36.082+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:36.459 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:36.082+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:36.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:36 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4[41789]: 2024-09-13T15:59:36.457+0000 7f131eba8640 -1 osd.4 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.455195+0000 front 2024-09-13T15:59:00.455217+0000 (oldest deadline 2024-09-13T15:59:25.055816+0000) 2024-09-13T15:59:37.366 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:37 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:37.064+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:37.366 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:37 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:37.064+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:37.367 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:37 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:37.283+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:37.367 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:37 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:37.283+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:37.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:37 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:37.365+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:37.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:37 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:37.365+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:38.096 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:38 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:38.039+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:38.096 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:38 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:38.039+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:38.096 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:37 smithi175 podman[72639]: 2024-09-13 15:59:37.806902531 +0000 UTC m=+5.814726008 container died c05efcd1485a378fc60ca9e68396f7a93270df02eb6df71e9a6c7414f106f583 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-13T15:59:38.369 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:38 smithi175 podman[72639]: 2024-09-13 15:59:38.19326148 +0000 UTC m=+6.201084956 container cleanup c05efcd1485a378fc60ca9e68396f7a93270df02eb6df71e9a6c7414f106f583 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-13T15:59:38.369 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:38 smithi175 bash[72639]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4 2024-09-13T15:59:38.369 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:38 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:38.326+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:38.369 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:38 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:38.326+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:38.765 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:38 smithi175 podman[72655]: 2024-09-13 15:59:38.46961398 +0000 UTC m=+0.661661509 container remove c05efcd1485a378fc60ca9e68396f7a93270df02eb6df71e9a6c7414f106f583 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-13T15:59:38.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:38 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:38.367+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:38.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:38 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:38.367+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:39.043 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:39 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:39.035+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:39.044 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:39 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:39.035+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:39.044 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:38 smithi175 podman[72744]: 2024-09-13 15:59:38.693362001 +0000 UTC m=+0.015943918 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:39.044 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:38 smithi175 podman[72744]: 2024-09-13 15:59:38.812064831 +0000 UTC m=+0.134646733 container create 39a29fadcc4716bc54f99e1fbb530332e815bef6b6329626dee05045425421df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:59:39.044 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:39 smithi175 podman[72744]: 2024-09-13 15:59:39.039711825 +0000 UTC m=+0.362293727 container init 39a29fadcc4716bc54f99e1fbb530332e815bef6b6329626dee05045425421df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, ceph=True) 2024-09-13T15:59:39.044 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:39 smithi175 podman[72744]: 2024-09-13 15:59:39.04320792 +0000 UTC m=+0.365789830 container start 39a29fadcc4716bc54f99e1fbb530332e815bef6b6329626dee05045425421df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:59:39.304 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:39 smithi175 podman[72744]: 2024-09-13 15:59:39.182014561 +0000 UTC m=+0.504596464 container attach 39a29fadcc4716bc54f99e1fbb530332e815bef6b6329626dee05045425421df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-13T15:59:39.304 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:39 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:39.282+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:39.304 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:39 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:39.282+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:39.712 INFO:journalctl@ceph.osd.4.smithi175.stdout:Sep 13 15:59:39 smithi175 podman[72744]: 2024-09-13 15:59:39.421862708 +0000 UTC m=+0.744444618 container died 39a29fadcc4716bc54f99e1fbb530332e815bef6b6329626dee05045425421df (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-13T15:59:39.713 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:39 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:39.373+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:39.713 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:39 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:39.373+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:40.065 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.4.service' 2024-09-13T15:59:40.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:40.104 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-13T15:59:40.104 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-13T15:59:40.105 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.5 2024-09-13T15:59:40.302 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:40.035+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:40.303 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:40.035+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:40.303 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:40 smithi175 systemd[1]: Stopping Ceph osd.5 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:40.303 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:40.250+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:40.303 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:40.250+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:40.765 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:40.455+0000 7f8ee4889640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:40.766 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:40.455+0000 7f8ee4889640 -1 osd.5 72 *** Got signal Terminated *** 2024-09-13T15:59:40.766 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:40.455+0000 7f8ee4889640 -1 osd.5 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:40.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:40.379+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:40.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:40 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:40.379+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:41.425 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:41 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:41.258+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:41.425 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:41 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:41.258+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:41.426 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:41 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:41.065+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:41.426 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:41 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:41.065+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:41.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:41 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:41.424+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:41.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:41 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:41.424+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:42.462 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:42.252+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:42.462 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:42.252+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:42.463 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:42.081+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:42.463 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:42.081+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:42.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:42.461+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:42.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:42 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:42.461+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:43.464 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:43 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:43.282+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:43.465 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:43 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:43.282+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:43.465 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:43 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:43.052+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:43.465 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:43 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:43.052+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:43.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:43 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:43.463+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:43.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:43 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:43.463+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:44.265 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:44.004+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:44.265 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:44.004+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:44.266 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:44.004+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:18.309867+0000 front 2024-09-13T15:59:18.309737+0000 (oldest deadline 2024-09-13T15:59:43.609840+0000) 2024-09-13T15:59:44.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:44.439+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:44.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:44.439+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:44.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:44.439+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:44.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:44.294+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:44.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:44.294+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:44.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:44 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:44.294+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:45.312 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:45.021+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:03.207387+0000 front 2024-09-13T15:59:03.207508+0000 (oldest deadline 2024-09-13T15:59:27.907456+0000) 2024-09-13T15:59:45.313 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:45.021+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.408628+0000 front 2024-09-13T15:59:11.408587+0000 (oldest deadline 2024-09-13T15:59:33.708307+0000) 2024-09-13T15:59:45.313 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5[47724]: 2024-09-13T15:59:45.021+0000 7f8edce95640 -1 osd.5 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:18.309867+0000 front 2024-09-13T15:59:18.309737+0000 (oldest deadline 2024-09-13T15:59:43.609840+0000) 2024-09-13T15:59:45.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:45.467+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:45.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:45.467+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:45.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:45.467+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:45.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:45.311+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:45.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:45.311+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:45.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:45 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:45.311+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:46.182 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:45 smithi175 podman[72858]: 2024-09-13 15:59:45.886226712 +0000 UTC m=+5.547006390 container died f733507daea46943b6bbdc50ed514fc868b9e3022f7296afd55408ccfc5b466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:46.508 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:46 smithi175 podman[72858]: 2024-09-13 15:59:46.297274897 +0000 UTC m=+5.958054574 container cleanup f733507daea46943b6bbdc50ed514fc868b9e3022f7296afd55408ccfc5b466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-13T15:59:46.508 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:46 smithi175 bash[72858]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5 2024-09-13T15:59:46.508 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:46 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:46.340+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:46.509 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:46 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:46.340+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:46.509 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:46 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:46.340+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:46.765 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:46 smithi175 podman[72875]: 2024-09-13 15:59:46.507372635 +0000 UTC m=+0.619031584 container remove f733507daea46943b6bbdc50ed514fc868b9e3022f7296afd55408ccfc5b466e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-13T15:59:46.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:46 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:46.511+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:46.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:46 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:46.511+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:46.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:46 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:46.511+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:47.219 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:46 smithi175 podman[72965]: 2024-09-13 15:59:46.916444616 +0000 UTC m=+0.110754761 container create 521ef4154a9e840b894dff7b490595a75c5afa3eac41cf6c4c93daa6a8bd3d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-13T15:59:47.219 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:46 smithi175 podman[72965]: 2024-09-13 15:59:46.821504415 +0000 UTC m=+0.015814528 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:47.220 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:47 smithi175 podman[72965]: 2024-09-13 15:59:47.143675805 +0000 UTC m=+0.337985903 container init 521ef4154a9e840b894dff7b490595a75c5afa3eac41cf6c4c93daa6a8bd3d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-13T15:59:47.220 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:47 smithi175 podman[72965]: 2024-09-13 15:59:47.146993462 +0000 UTC m=+0.341303579 container start 521ef4154a9e840b894dff7b490595a75c5afa3eac41cf6c4c93daa6a8bd3d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T15:59:47.220 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:47 smithi175 podman[72965]: 2024-09-13 15:59:47.218637212 +0000 UTC m=+0.412947330 container attach 521ef4154a9e840b894dff7b490595a75c5afa3eac41cf6c4c93daa6a8bd3d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:59:47.493 INFO:journalctl@ceph.osd.5.smithi175.stdout:Sep 13 15:59:47 smithi175 podman[72965]: 2024-09-13 15:59:47.340223859 +0000 UTC m=+0.534533969 container died 521ef4154a9e840b894dff7b490595a75c5afa3eac41cf6c4c93daa6a8bd3d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-13T15:59:47.493 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:47 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:47.338+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:47.494 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:47 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:47.338+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:47.494 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:47 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:47.338+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:47.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:47 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:47.491+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:47.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:47 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:47.491+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:47.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:47 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:47.491+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:47.954 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.5.service' 2024-09-13T15:59:47.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:47.995 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-13T15:59:47.995 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-13T15:59:47.995 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.6 2024-09-13T15:59:48.265 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 systemd[1]: Stopping Ceph osd.6 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:48.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:48.467+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:48.467+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:48.467+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:48.330+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:48.330+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:48.330+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:48.333+0000 7f491aa5b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:48.333+0000 7f491aa5b640 -1 osd.6 72 *** Got signal Terminated *** 2024-09-13T15:59:48.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:48 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:48.333+0000 7f491aa5b640 -1 osd.6 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:49.765 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:49 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:49.378+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:49.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:49 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:49.378+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:49.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:49 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:49.378+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:49.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:49 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:49.424+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:49.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:49 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:49.424+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:49.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:49 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:49.424+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:50.765 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:50 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:50.394+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:50.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:50 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:50.394+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:50.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:50 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:50.394+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:50.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:50 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:50.397+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:50.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:50 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:50.397+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:50.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:50 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:50.397+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:51.765 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:51.413+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:51.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:51.413+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:51.766 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:51.413+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:51.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:51.360+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:51.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:51.360+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:51.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:51.360+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:51.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:51 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:51.360+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:52.667 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:52.401+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:52.401+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:52.401+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:52.401+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:52.365+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:02.314658+0000 front 2024-09-13T15:59:02.314639+0000 (oldest deadline 2024-09-13T15:59:26.414643+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:52.365+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.016309+0000 front 2024-09-13T15:59:11.016327+0000 (oldest deadline 2024-09-13T15:59:35.116268+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:52.365+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:20.718490+0000 front 2024-09-13T15:59:20.718526+0000 (oldest deadline 2024-09-13T15:59:43.618473+0000) 2024-09-13T15:59:52.668 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:52 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6[53520]: 2024-09-13T15:59:52.365+0000 7f4913067640 -1 osd.6 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:28.720325+0000 front 2024-09-13T15:59:28.720291+0000 (oldest deadline 2024-09-13T15:59:52.120983+0000) 2024-09-13T15:59:53.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:53 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:53.372+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:53.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:53 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:53.372+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:53.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:53 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:53.372+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:53.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:53 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:53.372+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:54.076 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:53 smithi175 podman[73066]: 2024-09-13 15:59:53.779583438 +0000 UTC m=+5.560126186 container died 299ad23afa035ef10c138549e38418bf6204038517b0d07a715535ce9d38eb05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-13T15:59:54.393 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 podman[73066]: 2024-09-13 15:59:54.166847391 +0000 UTC m=+5.947390142 container cleanup 299ad23afa035ef10c138549e38418bf6204038517b0d07a715535ce9d38eb05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:59:54.394 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 bash[73066]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6 2024-09-13T15:59:54.752 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 podman[73094]: 2024-09-13 15:59:54.45985811 +0000 UTC m=+0.672975081 container remove 299ad23afa035ef10c138549e38418bf6204038517b0d07a715535ce9d38eb05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-13T15:59:54.753 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:54 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:54.392+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:54.753 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:54 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:54.392+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:54.753 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:54 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:54.392+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:54.753 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:54 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:54.392+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:55.016 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 podman[73184]: 2024-09-13 15:59:54.752294225 +0000 UTC m=+0.099200758 container create 6ca7492fc2e483f11bd67c097bf27a9754adfa2a962324351e95c328afa69ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-09-13T15:59:55.016 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 podman[73184]: 2024-09-13 15:59:54.66880363 +0000 UTC m=+0.015710175 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T15:59:55.016 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 podman[73184]: 2024-09-13 15:59:54.979647373 +0000 UTC m=+0.326553908 container init 6ca7492fc2e483f11bd67c097bf27a9754adfa2a962324351e95c328afa69ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6-deactivate, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-13T15:59:55.016 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:54 smithi175 podman[73184]: 2024-09-13 15:59:54.982933102 +0000 UTC m=+0.329839644 container start 6ca7492fc2e483f11bd67c097bf27a9754adfa2a962324351e95c328afa69ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T15:59:55.397 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:55 smithi175 podman[73184]: 2024-09-13 15:59:55.071254495 +0000 UTC m=+0.418161031 container attach 6ca7492fc2e483f11bd67c097bf27a9754adfa2a962324351e95c328afa69ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f) 2024-09-13T15:59:55.397 INFO:journalctl@ceph.osd.6.smithi175.stdout:Sep 13 15:59:55 smithi175 podman[73184]: 2024-09-13 15:59:55.171870261 +0000 UTC m=+0.518776795 container died 6ca7492fc2e483f11bd67c097bf27a9754adfa2a962324351e95c328afa69ad4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T15:59:55.758 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:55 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:55.395+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:55.759 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:55 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:55.395+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:55.759 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:55 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:55.395+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:55.759 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:55 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:55.395+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:55.777 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.6.service' 2024-09-13T15:59:55.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T15:59:55.818 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-13T15:59:55.818 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-13T15:59:55.818 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.7 2024-09-13T15:59:56.015 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:55 smithi175 systemd[1]: Stopping Ceph osd.7 for 41124e1c-71e4-11ef-bceb-c7b262605968... 2024-09-13T15:59:56.515 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.144+0000 7f814fed3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-13T15:59:56.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.144+0000 7f814fed3640 -1 osd.7 72 *** Got signal Terminated *** 2024-09-13T15:59:56.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.144+0000 7f814fed3640 -1 osd.7 72 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-13T15:59:56.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.364+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:56.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.364+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:56.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.364+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:56.516 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:56 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:56.364+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:57.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:57 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:57.324+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:57.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:57 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:57.324+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:57.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:57 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:57.324+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:57.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:57 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:57.324+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:58.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:58 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:58.352+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:58.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:58 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:58.352+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:58.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:58 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:58.352+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:58.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:58 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:58.352+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T15:59:59.765 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:59 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:59.336+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T15:59:59.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:59 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:59.336+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T15:59:59.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:59 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:59.336+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T15:59:59.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 15:59:59 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T15:59:59.336+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T16:00:00.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:00 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T16:00:00.329+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6806 osd.0 since back 2024-09-13T15:59:00.035983+0000 front 2024-09-13T15:59:00.036036+0000 (oldest deadline 2024-09-13T15:59:25.935940+0000) 2024-09-13T16:00:00.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:00 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T16:00:00.329+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6814 osd.1 since back 2024-09-13T15:59:11.236969+0000 front 2024-09-13T15:59:11.237130+0000 (oldest deadline 2024-09-13T15:59:34.737103+0000) 2024-09-13T16:00:00.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:00 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T16:00:00.329+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6822 osd.2 since back 2024-09-13T15:59:19.438261+0000 front 2024-09-13T15:59:19.438250+0000 (oldest deadline 2024-09-13T15:59:44.138169+0000) 2024-09-13T16:00:00.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:00 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T16:00:00.329+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.90:6830 osd.3 since back 2024-09-13T15:59:24.638948+0000 front 2024-09-13T15:59:24.638975+0000 (oldest deadline 2024-09-13T15:59:50.538950+0000) 2024-09-13T16:00:00.766 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:00 smithi175 ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7[59246]: 2024-09-13T16:00:00.329+0000 7f81484df640 -1 osd.7 72 heartbeat_check: no reply from 172.21.15.175:6804 osd.4 since back 2024-09-13T15:59:36.240690+0000 front 2024-09-13T15:59:36.240674+0000 (oldest deadline 2024-09-13T16:00:00.241098+0000) 2024-09-13T16:00:01.987 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:01 smithi175 podman[73287]: 2024-09-13 16:00:01.683068932 +0000 UTC m=+5.641861059 container died 773859a60aaf71c1736aac1e8069418ebf0ab7caed732d6e7bdb8f5394ce4b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-13T16:00:02.266 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 podman[73287]: 2024-09-13 16:00:02.076897566 +0000 UTC m=+6.035689690 container cleanup 773859a60aaf71c1736aac1e8069418ebf0ab7caed732d6e7bdb8f5394ce4b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-13T16:00:02.266 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 bash[73287]: ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7 2024-09-13T16:00:02.720 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 podman[73302]: 2024-09-13 16:00:02.369999849 +0000 UTC m=+0.681795693 container remove 773859a60aaf71c1736aac1e8069418ebf0ab7caed732d6e7bdb8f5394ce4b21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-13T16:00:03.016 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 podman[73389]: 2024-09-13 16:00:02.622676957 +0000 UTC m=+0.015668716 image pull 152d8bc6b19e354515e8ba109d4619445de0e09b4d4a747c2f16eb5c0edb670c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a 2024-09-13T16:00:03.016 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 podman[73389]: 2024-09-13 16:00:02.720371751 +0000 UTC m=+0.113363511 container create b13fca38724c0ab295a6238a9b52aa64d56b8ff7ad6551b0678bf7fc5a083621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-13T16:00:03.016 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 podman[73389]: 2024-09-13 16:00:02.948087693 +0000 UTC m=+0.341079452 container init b13fca38724c0ab295a6238a9b52aa64d56b8ff7ad6551b0678bf7fc5a083621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-09-13T16:00:03.016 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:02 smithi175 podman[73389]: 2024-09-13 16:00:02.951507559 +0000 UTC m=+0.344499328 container start b13fca38724c0ab295a6238a9b52aa64d56b8ff7ad6551b0678bf7fc5a083621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-13T16:00:03.402 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:03 smithi175 podman[73389]: 2024-09-13 16:00:03.06461834 +0000 UTC m=+0.457610100 container attach b13fca38724c0ab295a6238a9b52aa64d56b8ff7ad6551b0678bf7fc5a083621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7-deactivate, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-13T16:00:03.402 INFO:journalctl@ceph.osd.7.smithi175.stdout:Sep 13 16:00:03 smithi175 podman[73389]: 2024-09-13 16:00:03.133779282 +0000 UTC m=+0.526771041 container died b13fca38724c0ab295a6238a9b52aa64d56b8ff7ad6551b0678bf7fc5a083621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:195bcc5557a21b1fc5da9f42b2147dd7b8d9c3a0edf068f394db39472062247a, name=ceph-41124e1c-71e4-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240910.095512-reef-50bbe7f, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-13T16:00:03.682 DEBUG:teuthology.orchestra.run.smithi175:> sudo pkill -f 'journalctl -f -n 0 -u ceph-41124e1c-71e4-11ef-bceb-c7b262605968@osd.7.service' 2024-09-13T16:00:03.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-13T16:00:03.721 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-13T16:00:03.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T16:00:03.880 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T16:00:32.616 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-13T16:00:32.770 INFO:teuthology.orchestra.run.smithi175.stdout:Deleting cluster with fsid: 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T16:00:49.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T16:00:49.500 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-13T16:00:49.528 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-13T16:00:49.530 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734/remote/smithi090/crash 2024-09-13T16:00:49.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/crash -- . 2024-09-13T16:00:49.576 INFO:teuthology.orchestra.run.smithi090.stderr:tar: /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T16:00:49.576 INFO:teuthology.orchestra.run.smithi090.stderr:tar: Error is not recoverable: exiting now 2024-09-13T16:00:49.578 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734/remote/smithi175/crash 2024-09-13T16:00:49.579 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/crash -- . 2024-09-13T16:00:49.612 INFO:teuthology.orchestra.run.smithi175.stderr:tar: /var/lib/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-13T16:00:49.612 INFO:teuthology.orchestra.run.smithi175.stderr:tar: Error is not recoverable: exiting now 2024-09-13T16:00:49.613 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-13T16:00:49.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-09-13T16:00:49.646 INFO:teuthology.orchestra.run.smithi090.stderr:grep: /var/log/ceph/41124e1c-71e4-11ef-bceb-c7b262605968/ceph.log: No such file or directory 2024-09-13T16:00:49.649 INFO:tasks.cephadm:Compressing logs... 2024-09-13T16:00:49.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:00:49.693 DEBUG:teuthology.orchestra.run.smithi175:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:00:49.737 INFO:teuthology.orchestra.run.smithi090.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T16:00:49.761 INFO:teuthology.orchestra.run.smithi175.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-13T16:12:40.995 INFO:tasks.cephadm:Archiving logs... 2024-09-13T16:12:40.996 DEBUG:teuthology.misc:Transferring archived files from smithi090:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734/remote/smithi090/log 2024-09-13T16:12:40.997 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T16:13:52.895 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734/remote/smithi175/log 2024-09-13T16:13:52.896 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T16:14:52.519 INFO:tasks.cephadm:Removing cluster... 2024-09-13T16:14:52.520 DEBUG:teuthology.orchestra.run.smithi090:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 --force 2024-09-13T16:14:53.294 INFO:teuthology.orchestra.run.smithi090.stdout:Deleting cluster with fsid: 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T16:14:54.253 DEBUG:teuthology.orchestra.run.smithi175:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 41124e1c-71e4-11ef-bceb-c7b262605968 --force 2024-09-13T16:14:54.547 INFO:teuthology.orchestra.run.smithi175.stdout:Deleting cluster with fsid: 41124e1c-71e4-11ef-bceb-c7b262605968 2024-09-13T16:14:54.868 INFO:tasks.cephadm:Removing cephadm ... 2024-09-13T16:14:54.868 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T16:14:54.886 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-13T16:14:54.902 INFO:tasks.cephadm:Teardown complete 2024-09-13T16:14:54.902 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T16:14:54.941 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T16:14:54.942 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:14:54.944 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:14:55.048 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T16:14:55.057 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T16:14:55.058 DEBUG:teuthology.orchestra.run.smithi090:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:14:55.061 DEBUG:teuthology.orchestra.run.smithi175:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:14:55.073 INFO:teuthology.orchestra.run.smithi090.stderr:bash: line 1: ntpq: command not found 2024-09-13T16:14:55.075 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2024-09-13T16:14:55.102 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T16:14:55.102 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-09-13T16:14:55.102 INFO:teuthology.orchestra.run.smithi175.stdout:^* hv01.front.sepia.ceph.com 3 6 377 52 +279us[ +306us] +/- 37ms 2024-09-13T16:14:55.102 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 50 -280us[ -280us] +/- 54ms 2024-09-13T16:14:55.102 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -45us[ -45us] +/- 98ms 2024-09-13T16:14:55.102 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:14:55.158 INFO:teuthology.orchestra.run.smithi090.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T16:14:55.158 INFO:teuthology.orchestra.run.smithi090.stdout:=============================================================================== 2024-09-13T16:14:55.158 INFO:teuthology.orchestra.run.smithi090.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 +245us[ +239us] +/- 37ms 2024-09-13T16:14:55.158 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -307us[ -307us] +/- 54ms 2024-09-13T16:14:55.158 INFO:teuthology.orchestra.run.smithi090.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -83us[ -83us] +/- 99ms 2024-09-13T16:14:55.158 INFO:teuthology.orchestra.run.smithi090.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T16:14:55.183 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T16:14:55.193 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T16:14:55.194 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T16:14:55.202 DEBUG:teuthology.orchestra.run.smithi090:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T16:14:55.413 DEBUG:teuthology.orchestra.run.smithi175:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T16:14:55.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T16:14:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:14:55.620 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-13T16:14:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:14:55.650 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T16:14:55.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T16:14:56.327 INFO:teuthology.task.internal:Duration was 3386.767024 seconds 2024-09-13T16:14:56.327 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T16:14:56.398 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T16:14:56.398 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:14:56.401 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:14:56.462 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T16:14:56.476 INFO:teuthology.orchestra.run.smithi090.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T16:14:56.858 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T16:14:56.859 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi090.front.sepia.ceph.com 2024-09-13T16:14:56.859 DEBUG:teuthology.orchestra.run.smithi090:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:14:56.890 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi175.front.sepia.ceph.com 2024-09-13T16:14:56.890 DEBUG:teuthology.orchestra.run.smithi175:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:14:56.928 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T16:14:56.929 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:14:56.933 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:14:57.022 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T16:14:57.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:14:57.025 DEBUG:teuthology.orchestra.run.smithi175:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:14:57.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T16:14:57.440 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T16:14:57.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:14:57.473 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:14:57.499 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T16:14:57.514 DEBUG:teuthology.orchestra.run.smithi090:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:57.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:57.562 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = core 2024-09-13T16:14:57.565 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = core 2024-09-13T16:14:57.579 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:14:57.596 DEBUG:teuthology.orchestra.run.smithi175:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:14:57.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:14:57.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T16:14:57.655 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T16:14:57.656 DEBUG:teuthology.misc:Transferring archived files from smithi090:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734/remote/smithi090 2024-09-13T16:14:57.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:14:57.698 DEBUG:teuthology.misc:Transferring archived files from smithi175:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903734/remote/smithi175 2024-09-13T16:14:57.698 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:14:57.735 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T16:14:57.735 DEBUG:teuthology.orchestra.run.smithi090:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:14:57.737 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:14:57.784 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T16:14:57.860 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T16:14:57.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T16:14:57.874 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T16:14:57.875 DEBUG:teuthology.orchestra.run.smithi090:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:14:57.877 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:14:57.890 INFO:teuthology.orchestra.run.smithi090.stdout: 265571 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:14 /home/ubuntu/cephtest 2024-09-13T16:14:57.892 INFO:teuthology.orchestra.run.smithi175.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:14 /home/ubuntu/cephtest 2024-09-13T16:14:57.895 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T16:14:57.905 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T16:14:57.990 INFO:teuthology.run:Summary data: description: fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench conf/{client mds mgr mon osd} distro/single-container-host overrides/{ignorelist_health pg_health}} duration: 3386.7670238018036 flavor: default owner: scheduled_rishabh@teuthology success: true 2024-09-13T16:14:57.990 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:14:58.083 INFO:teuthology.run:pass